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In  the  last  few  years  many  aspects  of  computers  and 
computer  science  have  progressed  from  "black  art  mystery1^ 

a--'  jK 

to  a  precise  science.  One  of  the  "black  arts"  that  still 
remains  is  the  ability  to  correctly  estimate  the  cost  of 
upgrading  or  expanding  data  processing  operations# 
particularly  those  resulting  in  the  changing  of  vendors. 

The  federal  government  has  been  particularly 
plagued  by  gross  underestimation  of  large-scale  conversion 
costs  in  the  past.  Management  responsible  for  making 
sound#  economical  decisions  often  lacks  the  experience  in 
cost  estimation  needed  to  even  include  all  the  factors 
necessary  for  consideration  when  procuring  new  equipment. 
This  thesis  provides  a  guide  to  the  manager/analyst  who 


must  develop  a  cost  estimate  to  be  used  to  select  among 
competing  vendors.  Current  cost  estimation  techniques  are 
reviewed  within  the  framework  of  government  purchasing 
regulations  and  guidelines  affecting  computer  procurement. 
Separate  sections  discuss  intangible  factors  and  their 
importance  to  the  selection  process. 

This  background  sets  the  proper  perspective  for 
the  thesis'  case  study,  which  provides  the  basis  for  a 
post-implementation  development  of  a  list  of  intangible 
factors.  An  approach  is  also  developed  to  identify  and 
evaluate  intangible  factors t  this  suggested  approach  can 
be  tailored  and  applied  to  other  similar  computer  system 
purchases. 
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CHAPTER  I 


INTRODUCTION 

Man's  victories  over  nature,  in  science  and 
engineering,  are  closely  correlated  with  his 
ability  to  measure  whatever  he  is  working  with. 

This  is  related  to  the  ability  to  recognize  and 
measure  all  the  critical  factors  involved. 

[17,  p. 15] 

Change  seems  to  be  a  way  of  life  for  productive 
Data  Processing  (DP)  shops.  One  of  the  biggest  changes  is 
the  upgrading  or  replacement  of  the  computer  system. 

Often  the  mere  thought  of  a  computer  system  conversion 
"strikes  fear  in  the  heart  of  a  DP  manager."  [30,  p.13] 
Since  ignoring  the  need  for  a  new  system  is  not  a  viable 
option,  the  replacement  of  the  current  system  is  an 
inevitable  fact-of-life  that  must  be  planned.  Evaluating 
and  acquiring  a  new  computer  system  requires  a  clear 
picture  of  the  costs  involved.  The  selection  of  a 
cost-effective  computer  system  involves  the  ability  to 
identify  and  measure  the  critical  factors,  including  those 
factors  which  are  difficult  to  measure. 

The  Federal  government  uses  computers  in  virtually 
all  phases  of  its  operations,  with  the  Department  of 
Defense  (DOD)  being  the  "largest  consumer  of  computer 
hardware  and  services  in  the  government."  [24,  p.10] 


As  noted  by  the  Grace  Commission,  the  President's  Private 
Sector  Survey  on  Cost  Control,  rapid  advances  in  computer 
science  technology  coupled  with  the  time  consuming  Federal 
acquisition  process  have  resulted  in  a  state  of 
obsolescence  for  many  DOD  general  purpose  computer 
installations.  [24,  pp. 10-11]  Since  computer  technology 
is  often  critical  to  the  ability  of  the  Services  to 
perform  their  missions,  it  is  mandatory  that  these 
obsolete  systems  be  replaced. 

Although  the  government's  record  for  handling 
large-scale  conversions  in  the  past  has  been  reported  by 
the  General  Accounting  Office  (GAO)  as  "dismal”,  with 
final  costs  often  exceeding  estimations  by  factors  of  ten, 
the  GAO  is  optimistic  about  future  conversion  activities. 
GAO  feels  that  "the  root  cause  of  the  problems  identified 
was  lack  of  adequate  planning  and  management  control..." 
and  that  "if  all  conversion  requirements  are  considered 
and  planned  for  ...  agency  officials  should  be  able  to 
make  sound,  economical  decisions  about  which  vendor  to 
choose...".  [23,  p.17] 

Statement  of  the  Problem 

Selecting  the  most  economical  computer  for  the  job 


has  become  increasingly  complex  with  the  rapid 


technological  advances  in  computers#  the  vast  variety  of 
equipment#  and  the  multitude  of  combinations  and  possible 
mixes  of  different  equipment.  To  evaluate  the  acquisition 
of  a  computer  system#  one  needs  a  clear  picture  of  the 
costs  involved.  Developing  this  clear  picture  of  costs  is 
difficult  even  for  specialists  in  the  field  and  could  "... 
require  a  full-time  staff  studying  models  and  makes  of 
available  equipment."  [30#  p.51] 

The  problem  of  replacement  of  DOD  computer 
equipment  is  further  complicated  by  complex  and  sometimes 
conflicting  regulations#  policies#  and  guidelines 
governing  federal  Automatic  Data  Processing  (ADP) 
acquisition.  Funds  are  often  not  available  to  hire 
outside  consultants#  and  even  when  outside  consultants  are 
used#  "in-house  personnel  will  be  heavily  involved"  in 
planning#  developing,  and  testing  during  the  different 
phases  of  acquiring  a  new  system.  [23#  p.17] 

The  policy  of  enhancing  the  competition  for  a 
government  contract  to  produce  the  best  cost  savings  makes 
the  possibility  of  each  computer  replacement  resulting  in 
a  change  of  computer  vendors  a  reality.  This  possible 
change  of  vendors  needs  to  be  analyzed#  using  proper  life 
cycle  management  principles#  to  select  the  computer  system 
that  will  really  be  the  most  economical.  The  key  to  the 
development  of  a  correct  life  cycle  cost  for  each  vendor, 


•  •  • 


IS 


the  identification  of  all  the  cost  drivers  germaine 


to  the  acquisition."  124,  p.12]  If  these  cost  drivers  are 
not  all  identified  or  if  they  are  not  correctly  analyzed, 
then  the  lowest  bid  computer  system  could  turn  out  to  be 
the  most  costly  choice  in  reality. 

This  thesis  addresses  the  problem  of  correctly 
identifying  and  estimating  all  the  costs  of  computer 
system  replacements,  especially  those  costs  which  are 
classified  as  intangible  and  are  especially  difficult  to 
include  in  system  life  cycle  costs. 


Objectives  and  Scope 

The  overall  objectives  of  this  thesis  are 
threefold.  The  first  objective  is  to  document  the  existing 
ADP  acquisition  environment  and  its  application  that 
results  in  the  conversion  of  large  scale  computer  systems 
to  new  vendors  when  DOD  systems  are  upgraded  or  replaced. 
The  second  objective  is  to  present  a  review  of  current 
cost  estimation  techniques  that  may  be  used  at  the  vendor 
selection  stage  of  computer  system  acquisition.  Perhaps 
the  most  difficult  factors  to  include  in  these  analyses 
are  intangible  factors,  so  the  third  objective  is  to 
develop  an  approach  to  measure  intangible  costs  and 
benefits.  The  third  objective  includes  the  development  of 
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an  intangible  factor  profile ,  which  was  developed  by 
studying  an  actual  DOD  large  computer  system  acquisition 
that  resulted  in  a  change  of  vendors.  This  profile  could 
serve  as  a  guide  to  future  research  into  developing  the 
best  possible  estimation  of  costs  to  evaluate  the 
proposals  of  competing  vendors. 

Specific  activities  undertaken  for  this  study 

were: 

-  Reviewing  existing  regulatory/policy  guidance 
that  directs  or  influences  the  acquisition 
process. 

-  Reviewing  current  literature  for  cost  factors 
and  techniques  of  cost  estimation  applicable  to 
a  large  system  replacement. 

-  Identification,  evaluation,  and  inclusion  of 
intangible  factors  in  the  cost  estimation 
process. 

-  Interviews  with  key  personnel  to  develop  case 
study  data. 

-  Reviewing  key  documents  associated  with  the 
case  study's  history  and  current  status. 

-  Formulation  and  validation  of  a  Intangible 
Factor  Profile  using  case  study  data. 

-  Suggestion  of  possible  improvements. 

-  Suggestion  of  possible  areas  for  future  study. 


This  study  is  limited  to  considering  only  those 
non-tactical  DOD  computer  acquisitions  and  excludes  the 
category  of  computers  that  is  considered  critical  to 
direct  fulfillment  of  military  or  intelligence  missions, 
such  as  special  embedded  weapon  systems'  computers.  These 
special  systems  are  not  subject  to  the  same  guidelines  and 
directives  as  the  non-tactical  systems.  Non-tactical 
systems  perform  functions  much  like  general  purpose 
business  applications.  This  study  is  further  limited  to 
the  investigation  of  cost  estimation  when  procuring  a 
large  computer  system,  or  mainframe;  while  some  of  the 
techniques  may  be  applicable  to  the  acquisition  of  mini- 
or  micro-computer 8,  that  is  not  the  direction  of  the 
study. 


While  private  sector  companies  do  not  directly 
parallel  the  DOD  use  of  computer  systems,  there  is  still 
enough  similarity  in  their  use  and  business  practices  to 
assume  that  findings  regarding  computer  costs  in  the 
private  sector  will  be  applicable  to  non-tactical  DOD 


operations.  It  is  also  assumed  that  federal  guidelines 
and  policies  drive  some  unique  differences  between  DOD 
acquisitions,  and  private  sector  acquisitions  that  must  be 
addressed. 

Additionally,  it  is  assumed  that  the  inclusion  of 
intangible  factors  in  a  cost  comparison  is  highly 
desirable  and  achievable,  with  some  degree  of  accuracy. 


This  thesis  addresses  the  estimation  of  cost  for 
competing  vendors  in  a  large  computer  ByBtem  replacement, 
specif icly  a  non-tactical  DOD  acquisition.  Chapter  II 
sets  the  background  for  federal  acquisition  of  ADP 
equipment  by  briefly  presenting  a  historical  background  on 
federal  acquisition,  covers  regulations,  guidelines  and 
policies,  and  outlines  the  acquisition  process.  Chapter 
III  reviews  several  current  cost  estimation  techniques 
used  for  vendor  selection  found  in  literature.  Chapter  IV 
provides  the  same  type  of  review  for  the  estimation  of 
intangible  costs  and  benefits.  Chapter  V  presents  the 
case  study  of  a  large  DOD  computer  replacement  invovling  a 
change  of  vendors,  which  serves  as  an  post-installation 
method  of  developing  and  validating  an  intangible  factor 
profile.  Results  of  the  profile  are  also  interpreted  in 


this  chapter.  The  final  chapter  summarizes  the 
significant  findings  and  potential  uses  for  the  estimation 
techniques  and  recommends  future  research  directions. 


CHAPTER  II 


ADP  ACQUISITION  POLICY  AND  PROCESS 

The  basic  objectives  of  government  and  civilian 
industrial  purchasing  are  similar  in  that  both  are 
supporting  operations  and  trying  to  buy  competitively  and 
wisely  from  reliable  sources.  Government  purchasing  is 
enough  different  from  industrial  purchasing  to  warrant  at 
least  a  brief  examination  of  those  differences. 

The  primary  difference  is  the  technical  nature  of 
many  of  the  products  being  purchased.  Additionally,  the 
source  of  funds  used  for  purchases  neccessitates  the  use 
of  more  constraining  procedures  designed  to  protect  the 
interests  of  the  taxpayers.  These  government  procedures 
"...  stem  from  specific  laws  and  regulations  which  require 
competitive  bidding,  fixed  budgetary  limitations,  rigid 
auditing  of  accounts,  and  the  use  of  prescribed  standard 
specifications."  These  procedures  generally  "...  allow 
government  purchasing  managers  considerably  less  freedom 
of  action  and  discretion  than  business  allows  their 
industrial  counterparts."  [12,  pp. 634-5] 

Additionally,  when  comparing  "the  nation's  largest 
buyer"  to  industry,  two  facts  must  be  remembered.  First, 
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the  responsibility  of  defending  the  nation  often  makes 
minimizing  costs  a  trivial  issue.  Second,  our  government 
strives  for  joint  goals  of  equity  and  efficiency.  The  two 
goals  are  often  conflicting:  efficient  solutions  are 
frequently  regarded  as  unfair  and  "...equitable  solutions 
are  often  regarded  by  many  as  hopelessly  inefficient." 
Industry,  on  the  other  hand,  does  not  have  to  address  this 
goal  of  equity  and  can  "...  march  to  the  drummer  of 
efficiency."  [12,  p.670] 

Historical  Perspective 

Laws  regulating  government  procurement  date  back 
to  the  Second  Congress  in  1792.  Early  congressional 
interest  in  the  excessive  profits  of  contractors  and 
procurement  abuses  resulted  in  the  1808  law  requiring  a 
clause  in  every  government  contract  that  "...  no  member  of 
Congress  might  benefit  therefrom."  [12,  p.635]  Graft  and 
favoritism  continued,  so  Congress  passed  the  Procurement 
Act  of  1809  to  require  that  government  purchases  be  made 
using  formal  advertising  and  competitive  bidding.  [12, 
p.636] 

For  the  next  fifty  years,  formal  advertising  and 
competitive  bidding  was  further  defined  and  expanded. 
Competitive  bidding  continued  to  serve  the  government  well 


for  over  a  hundred  years,  in  fact  today's  Congressional 
mandate  continues  to  be  to  maximize  competitiion  and 
accountability.  However,  during  World  Wars  I  and  II,  the 
volume  of  purchases  and  the  technological  complexity  of 
the  purchases  caused  the  breakdown  of  the  old  competitive 
process.  Congress  recognized  the  necessity  of  negotiated 
purchasing  with  passage  of  the  Armed  Services  Procurement 
Act  of  1947.  [24,  p.18] 

Since  the  1930s  depression,  the  government  has 
also  tried  to  promote  socio-economic  issues  through 
procurement  policies,  such  as:  minimum  wage,  "Buy 
America",  age-sex  discrimination,  small  business 
preference,  labor  surplus  preference,  and  environmental 
pollution.  [24,  pp. 19-20]  The  final  result  has  been  a 
complex,  confusing,  and  sometimes  contradictory  collection 
of  statutes,  executive  orders,  regulations,  guidelines, 
policies,  and  processes. 

After  a  review  of  all  facets  of  government 
procurement  in  1972,  Congress  established  the  Office  of 
Federal  Procurement  Policy  (OFPP)  within  the  Office  of 
Management  and  Budget.  The  OFPP's  mandate  is  ”...  to 
develop  a  uniform  procurement  system  for  the  federal 
government,  giving  consideration  to  the  dissimilar  program 
activities  of  the  executive  agencies."  The  impact  of 


these  recent  developments  can  not  be  measured  yet.  [12, 
p.647] 
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The  ADP  regulatory  environment  is  comprised  of: 
public  laws,  executive  orders.  Office  of  Management  of 
Budget  (OMB)  circulars,  Federal  procurement  regulations. 
Office  of  Federal  Procurement  Policy  (OFPP)  publications. 
Federal  information  processing  standards.  Department  of 
Defense  (DOD)  directives  and  instructions,  and  the 
instructions  of  different  branches  of  service.  [24,  p.20] 
All  must  be  understood  and  applied  when  making  a 
government  purchase  of  ADP  equipment,  but  this  paper 
presents  only  a  few  of  the  most  significant  ones. 

The  basic  law  governing  DOD  procurement  is  the 
Armed  Services  Procurement  Act  of  1947,  which  established 
workable  policies  for  procurement  during  national 
emergencies  and  recognized  the  method  of  negotiated 
procurement  under  special  circumstances.  It  was  clarified 
in  the  Armed  Services  Procurement  Regulations  (ASPR)  which 
"...  governed  military  procurement,  set  limitations  on  the 
use  of  certain  types  of  contracts,  and  emphasized  the 
importance  of  small  business  participation  in  government 
contracting."  [24,  p.19]  The  ASPR  is  now  known  as  the 
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Defense  Acquisition  Regulation  (DAR) . 

Another  basic  procurement  law  is  the  Federal 
Property  and  Administrative  Services  Act  of  1949.  This 
act  is  directed  primarily  toward  civilian  agencies;  the 
GSA  issued  the  civilian  version  of  the  DAR,  the  Federal 
Procurement  Regulations ,  under  the  authority  of  this  act. 
Certain  sections  of  these  regulations ,  which  include  data 
processing  and  ADP  acquisition,  are  also  mandatory  for  DOD 
use.  [24,  pp.20,25] 

One  of  the  most  important  and  well-known  laws 
affecting  ADP  resources  is  Public  Law  89-306,  the  Brooks 
Act.  The  Brooks  Act  was  based  on  the  principles  that  "... 
ADP  resources  should  be  procured  as  economically  and 
efficiently  as  possible;  and  only  those  resources  should 
be  procured  which  are  needed  and  which  can  assist  the 
management  of  government  programs."  [27,  p.9]  The  bill 
gave  the  General  Services  Administration  (GSA)  "... 
authority  to  acquire,  operate,  fund,  and  dispose  of  ADPE 
for  the  entire  Federal  Government."  [9,  p.13]  GSA  was  not 
to  determine  ADPE  requirements  for  the  individual 
agencies. 

Over  the  years  the  close  review  of  ADPE 
acquisitions  by  Congressman  Brooks'  House  Government 
Operations  Committee  has  pressured  GSA  to  insure  maximum 
competition  is  possible  for  ADPE  contracts.  This  "... 
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over-riding  requirement  for  maximum  competition..."  has 
forced  elimination  of  cost  cosiderations,  like  software 
conversion,  when  making  proposals  and  evaluating  vendors. 
[9,  p.13]  The  DOD  has  sought  exemption  from  the  act  and 
received  partial  exemption  in  the  1982  Defense 
Authorization  Act,  but  mainly  for  equipment  and  services 
for  intelligence,  cryptology,  command  and  control,  or 
weapon  systems.  [27,  p.12] 

The  1980  Paperwork  Reduction  Act  gave  OMB  the 
authority  to  draft  government-wide  systems  acquisition 
policy  and  established  the  Office  of  Information  and 
Regulatory  Affairs  "...  as  the  focal  point  for  leadership 
and  central  direction  of  Federal  information  resource 
management.  [24,  p.22]  The  1985  Competition  in 
Contracting  Act  "...  requires  the  use  of  full  and  open 
competition  as  the  primary  method  of  procurement."  [27, 
p.12]  In  April  of  1985  the  new  unified  Federal 
Acquisition  Regulation  became  "...  the  basic  set  of 
procurement  rules  for  virtually  all  federal  agencies' 
acquisition  of  equipment,  supplies,  and  services."  [5, 
p.  32] 

Finally,  each  of  the  Services  has  its  own  specific 
sets  of  instructions  or  regulations  guiding  acquisition, 
management,  and  utilization  of  ADP  resources.  Presentation 
of  each  specific  branch's  regulations  is  outside  the  scope 


of  this  general  familiarization  with  government 
purchasing,  although  Chapter  5  mentions  some  specifics  in 
the  case  study. 


Guidelines  and  Policies 

There  is  perhaps  a  subtle  difference  between  a 
law,  or  regulation,  and  guidelines  and  policies. 
Guidelines  and  policies  tend  to  clarify  the  intent  of  the 
law  and  to  specify  implementation.  The  guidelines  and 
policies  for  ADP  acquisition  are  grouped  for  study  by  the 
agency  which  issued  them.  The  hierarchy  is  similar  to 
that  used  for  regulations,  and  again  omits  specifics  for 
the  different  branches  of  service,  although  such 
guidelines  do  exist. 

Congressional  committees  have  become  increasingly 
involved  in  monitoring  ADP  acquisition  and  use  since  the 
passage  of  the  Brooks  Act.  Federal  agencies  must  submit 
proposed  budgets,  major  policy,  and  guidance  to  the 
appropriate  committees  for  review.  The  committees  with 
legislative  and  oversight  authority  for  ADP  are: 

-  House  Committee  on  Government  Affairs  (HCOGO) 

-  Senate  Committee  on  Government  Affairs  (SCOGO) 

-  House  and  Senate  Armed  Services  Committee  (HASC  and 
SASC) 

-  House  and  Senate  Appropriations  Committee  (HAC  and 
SAC)  (24,  p. 32] 


.WITOBWKWLVW:  V*  .  W  *7  'TT <*.  ?  V  •’ 


,.  .v  .  ).,.v  n r»  «."  ■'.-  V  ',-c.v'-  ».■ ,.v  V- 


16 

Congressional  opinion  that  the  executive  agencies 
have  exercised  poor  management  of  ADP  resources  and 
ineffective  implementation  of  PL  89-306  has  resulted  in 
very  close  scrutinity  and  oversight  of  ADP  acquisition. 
Unfortunately ,  ADP  acquisition  policy  stressed  by  the 
different  committees  has  not  necessarily  been  the  same 
over  the  years.  HCOGO  has  stressed  the  policy  of 
maximizing  hardware  competition,  while  HAC  and  SASC  has 
championed  the  policy  of  lowest  total  life-cycle  cost. 
Perhaps  the  most  powerful  influence  has  been  the  HCOGO 
with  its  management  influence  through  GSA.  [24,  p.32] 

Executive  orders  direct  organizational  roles  and 
responsibilities.  Policy  functions  have  been  transferred 
from  OHB  to  GSA  and  back  again  through  Executive  Orders 
11717  and  11893,  in  1973  and  1975.  OMB  is  responsible  for 
policy  oversight  and  formulation,  while  GSA's  role  is  to 
develop  policy.  In  1982  Executive  Order  12352  introduced 
a  third  organization,  the  Office  of  Federal  Procurement 
Policy  (OFPP) ,  into  ADP  policy  formulation  and  control. 

Although  OFPP  was  established  within  OMB  to  "... 
provide  overall  direction  of  procurement  policies, 
regulations,  procedures,  and  forms",  it  allowed  GSA  to 
continue  responsibility  for  ADP  acquisition  policy.  OFPP 
instead  turned  its  efforts  toward  the  development  of  the 
new  Federal  Acquisition  Regulations  (FAR)  which  were  to 


consolidate  DOD,  GSA,  and  NASA  common  procurement 
regulations  into  a  simpler,  unified  system.  OFPP  also 
publishes  guidance  pamphlets.  [24,  pp. 23, 26, 33] 

OMB  communicates  policy  and  procedural  guidance 
through  its  circulars.  In  1965  OMB  Circular  A-71 
identified  GSA's  responsibilities  in  achieving  increased 
cost  effectiveness  in  ADPE  selection,  acquisition,  and 
utilization.  Guidance  for  preparation  and  submission  of 
annual  budgets  can  be  found  in  OMB  Circular  A-ll,  1978. 

OMB  Circular  A-76,  1979,  sets  forth  policies  for  the 
government's  acquisition  of  commercial  or  industrial 
products  (not  limited  to  ADP) .  Specific  guidance  was 
provided  for  major  ADP  acquisitions  by  Federal  agencies  in 
OMB  Circular  A-109,  1976.  Implementation  of  these 
procedures  has  been  difficult,  as  it  conflicts  with 
certain  provisions  of  the  Brooks  Act  and  GSA  has  not 
provided  specific  implementation  guidance.  [24,  pp.24-5] 
Under  the  Brooks  Act  the  National  Bureau  of 
Standards  (NBS),  within  the  Department  of  Commerce,  was 
tasked  with  providing  ADP  scientific  and  technical  advice 
to  federal  agencies  and  to  establish  ADP  standards.  NBS 
issues  many  publications  relating  to  numerous  functions  in 
ADP,  such  as  "...benchmarking,  management  of  multi-vendor 
plug-compatible  systems,  standardization,  and  security." 
[24,  pp.21,26] 


DOD  directives,  instructions,  and  manuals  provide 
further  guidance  in  implementing  statutory  provisions  and 
agencies'  policies.  By  1985  there  were  37  different 
publications  available  on  ADP  acquisition  and  utilization. 
To  name  a  few:  DOD  Directive  5000.1  and  Instruction 
5000.2  implement  OMB  Circular  A-109  within  DOD  primarily 
for  major  systems  acquisitions;  Directive  5100.40 
established  the  Assistant  Secretary  of  Defense 
(Comptroller)  as  the  Senior  ADP  Policy  Official;  Directive 
7920.1  established  technical  and  functional  policy 
concerning  life  cycle  management  of  ADP  equipment; 
Instruction  7920.2  supplements  7920.1  and  presents 
processes  and  procedures  for  review  and  decision  making 
during  the  approval  process.  (24,  pp.26-8] 


Government  acquisition  is  similiar  to  industrial 
acquisition  in  that  first  a  need  for  the  item  must  be 
recognized  and  justified.  Next  the  appropriate  level  of 
authority  must  approve  the  need  and  authorize  the 
expenditure  for  the  item,  which  means  enough  money  must  be 
available  in  the  budget.  Finally,  some  method  must  be 
used  to  select  and  purchase  the  item. 

Government  acquisition  is  most  unlike  industrial 
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acquisition  in  its  lack  of  flexibility.  Government 
purchases  usually  require  layers  of  approvals ,  are 
governed  by  a  myriad  of  rules,  and  must  often  use 
restrictive  methods  of  purchasing.  Studies  by  the  General 
Accounting  Office  have  described  the  current  acquisition 
cycle  as  "...  long,  complicated  and  frustrating  ...  a 
major  contributor  to  the  obsolescence  of  federal 
computers.”  [9,  p. 6]  Major  acquisitions  within  the 
services  can  average  over  ten  years,  while  non-federal 
complex  acquisitions  average  under  two  years.  [16,  p.2] 

To  better  understand  why  the  federal  ADP 
acquisition  process  can  be  so  lengthy,  it  is  necessary  to 
know  something  about  approval  levels,  types  and  methods  of 
purchasing,  and  selection  of  sources.  Additionally,  it  is 
helpful  to  know  which  of  these  methods  is  favored. 

All  potential  acquisitions  must  be  prepared  and 
submitted  with  justification  up  the  chain  of  command  for 
approval.  The  level  to  which  submissions  must  go  is 
related  to  the  estimated  expenditure.  In  the  case  of  a 
major  system  acquisition,  control  can  be  traced  from  GSA, 
through  the  Comptroller,  to  the  Senior  Policy  Official  for 
the  service,  on  down  through  the  rest  of  the  services' 
chain  of  command,  until  finally  reaching  the  agency 
desiring  the  new  system.  [24,  p.37]  Such  submissions  and 
approvals  involve  time  to  prepare  at  the  initiating  level 


and  to  evaluate  at  all  the  higher  levels. 

If  the  acquisition  is  approved,  then  the  actual 
purchase  steps  can  be  initiated.  In  the  case  of  a  major 
system  acquisition  the  money  will  have  to  be  budgeted 
years  in  advance  of  the  actual  purchase;  small  ADP 
acquisitions  can  be  made  much  faster.  GSA  maintains  many 
open-end  contracts  from  which  small  items:  printers, 
software  packages,  micro-computers,  and  the  like;  may  be 
obtained  with  a  government  purchase  order.  These 
contracts  are  usually  the  result  of  nationwide  advertising 
and  competitive  bidding.  [12,  p.642] 

In  the  case  of  a  major  system  purchase,  a 
Delegation  of  Procurement  Authority  (DPA)  would  be 
obtained  from  GSA.  The  Brooks  Act  made  GSA  the  sole 
procurement  authority  for  ADP  resources;  more 
specifically,  the  Office  of  Information  Management  within 
GSA,  formerly  known  as  the  Automated  Data  and 
Telecommunications  Service,  supervises  ADP  procurement. 
Since  GSA  is  not  staffed  to  make  the  actual  procurement, 
most  of  the  ADP  acquisitions  are  made  by  the  requesting 
agency  via  a  DPA.  The  wording  of  the  DPA  controls  and 
limits  the  types  of  systems,  methods,  and  costs  considered 
by  the  requesting  agency.  The  GSA  monitors  the  requesting 
agencies  compliance  with  the  DPA's  terms,  and  any 
violations  or  variances  from  the  DPA's  intent  could  result 
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in  GSA  revoking  DPA  authority.  [9,  p.13] 

Three  different  methods  of  purchasing  may  be  used 
by  the  government:  formal  advertising,  procurement  by 
negotiation,  and  sole-source  negotiation.  Roughly  90  to 
92  percent  of  the  total  dollar  value  of  military 
procurement  is  spent  through  the  negotiation  process;  the 
remaining  8  to  10  percent  is  spent  through  formal 
advertising.  "  'Advertised  bidding*  and  'negotiation' 
have  specialized  meanings  in  government  purchasing.  A 
knowledge  of  these  meanings  is  essential  to  understanding 
government  purchasing."  [12,  p.647] 

Formal  advertised  bidding  involves  five  steps. 
First  is  the  preparation  of  an  invitation  for  bid  (IFB) , 
which  is  ”...  a  complete  purchasing  package,  including  all 
contractural  requirements  and  terms."  Second,  the  IFB  is 
distributed  to  a  large  number  of  bidders.  Third  is  the 
public  opening,  reading,  and  recording  of  the  bids;  no 
bids  may  be  revised  or  withdrawn  at  this  point.  Fourth, 
those  bidders  not  conforming  precisely  to  the  IFB  are 
eliminated;  bidders  proven  to  be  flagrantly 
nonresponsible  also  can  be  eliminated.  Finally,  the 
contract  is  awarded  to  "...  that  resposnsibl[  bidder  whose 
price  is  lowest,  provided  it  is  deemed  reasonable  and  most 
advantageous  to  the  government,  all  factors  considered." 
[12,  p.647] 
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Procurement  by  negotiation  is  much  more  flexible. 
Vendors  are  selected  to  receive  invitations  to  bid,  and 
bids  may  be  revised  and  resubmitted  during  negotiations. 
Neither  the  information  about  the  bidding  nor  number  and 
identity  of  the  bidders  is  made  public.  The  contracting 
officer  has  much  more  freedom  in  negotiated  bidding  and 
acts  more  like  an  industrial  buyer.  Negotiated  purchasing 
is  permitted  under  the  Armed  Services  Procurement  Act; 
this  act  lists  seventeen  exceptions  that  allow  negotiation 
rather  than  formal  advertising  to  be  used.  [12, 
pp. 649-652] 

Sole-source  negotiation  involves  a  single 
supplier;  it  usually  starts  as  competitive  negotiation 
from  which  the  most  competent  supplier  is  chosen.  This 
form  of  negotiation  is  usually  reserved  for  "...  high 
dollar  contracts  for  major  weapon  systems  ...”, 
representing  75  percent  of  the  total  defense  dollar.  This 
form  of  purchasing  may  also  be  appropriate  for  the 
tactical  type  of  ADP  purchases  mentioned  previously. 

About  50  percent  of  defense  purchasing  is  sole-source  due 
to  the  unique  nature  of  the  products  being  bought; 
nevertheless,  effective  competition  is  ”...  the  most 
practical  single  means  of  establishing  a  reasonable 
price."  [12,  p.652-3] 

To  summarize  the  difference  between  government  and 
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industrial  methods,  industry  uses  competitive  negotiating 
more  frequently  and  has  the  freedom  to  use  whatever  method 
seems  most  profitable. 

Regardless  of  the  financial  and  business 
advantages  . . .  Congress  simply  does  not  seem  to  favor 
negotiated  procurement.  Most  government  contracting 
officers  ...  use  formal  advertised  procurement 
whenever  possible.  This  approach  to  selecting 
purchasing  methods  is  in  marked  contrast  to  industrial 
purchasing  practice  and  sound  purchasing  theory. 

[12,  p.657] 

Selection  of  sources  is  another  area  with  less 
freedom  for  the  government  buyer.  "The  industrial  buyer 
is  free  to  choose  suppliers  on  the  basis  of  total  value", 
which  includes  such  factors  as  quality,  dependability, 
service,  and  attitude  toward  customers.  With  formal 
advertising  the  source  is  selected  automatically  with  the 
lowest  bid.  Negotiated  procurement  seems  to  allow  the 
contracting  officer  to  select  the  best  suppliers,  but  in 
reality  is  "...  directly  restricted  by  legislation  and 
indirectly  restricted  by  various  procedural  requirements.” 
[12,  p.657] 

Most  of  these  restricting  laws  have  socio-economic 
goals:  Fair  Employment  Practices  Act,  Buy  America  Act, 
Equal  Employment  and  Opportunity  Executive  Order,  Labor 
Surplus  Program,  and  Small  Business  Act.  The  Small 
Business  Act  provides  a  good  example  of  the  impact  of 
these  socio-economic  laws.  Procurement  requirements, 


deemed  to  have  small  business  applications,  can  either  be 
totally  set  aside  for  small  businesses  or  divided  into 
portions  for  big  and  small  businesses.  The  intent  of  this 
law  is  admirable,  to  see  that  small  businesses  receive  a 
fair  share  of  the  government's  procurement  dollar.  But, 
the  result  is  that  "...  prices  paid  to  small  business 
firms  appear  to  be  substantially  higher  than  those  paid  to 
big  business  firms  for  similar  purchases."  [12,  pp. 658-9] 
The  bottom  line  of  this  brief  review  of  the 
different  factors  affecting  the  government  acquisition 
process  is  that  government  purchasing  "...  frequently 
involves  special  considerations  which  usually  are  not 
applicable  to  purchasing  in  private  industry.”  [12,  p.634] 
This  author's  reaction  to  a  review  of  the  maze  of  laws, 
regulations,  policies,  guidelines,  and  preferences  is 
sheer  wonder  that  business  is  ever  conducted  1 


CHAPTER  III 


CURRENT  METHODS  OF  VENDOR  SELECTION 

The  preceding  chapter  describes  the  complex  and 
confusing  regulatory  environment  that  promotes  competitive 
government  purchasing/  which  often  results  in  the  change 
of  computer  vendors  when  systems  are  upgraded  or  replaced. 
Although  the  emphasis  is  on  competition/  the  government 
still  wants  to  purchase  the  most  economical  computer 
system.  To  select  the  most  economical  system/  the 
competing  systems  must  be  compared  in  some  sort  of 
systematized  manner  to  select  the  vendor  that  will  provide 
the  lowest  life-cycle  costs  and  the  greatest  value. 

The  objective  of  this  chapter  is  to  provide  a 
review  of  several  methods/  currently  recommended  in 
literature/  that  may  be  used  to  compare  and  select  among 
different  vendors.  (It  is  assumed  that/  at  this  stagef 
vendors  who  do  not  fill  mandatory  requirements  have 
already  been  eliminated.)  A  special  area  of  interest  for 
this  author  is  the  inclusion  of  intangible  costs  and 
benefits  into  the  selection  process;  a  separate  chapter 
on  this  topic  follows  the  descriptions  of  current 
selection  methods.  These  different  methods  were  selected 
by  the  author  on  the  criterion  of  suitability  for  use  to 


select  a  large  computer  system  replacement. 

Six  methods  used  for  vendor  selection  were  chosen 
by  the  author  for  presentation: 

(1)  Cost/Benefit  Analysis 

(2)  Weighted  Score 

(3)  Payoff  Matrix 

(4)  Cost-Value 

(5)  Interactive  Financial  Planning  System 

(6)  Weighted  Evaluation  of  Cost  Factors 

The  first  two  methods  are  the  best  known  and  most 
widely  used.  Cost/Benefit  Analysis  could  be  considered 
the  "classic"  and  its  accounting  method  approach  is  very 
familiar  to  management.  The  Weighted  Score  method  is 
often  used  for  smaller  purchases  because  of  its 
simplicity,  but  has  a  problem  with  the  subjectivity 
introduced  into  the  decision  when  determining  the  weights. 

The  Payoff  Matrix  is,  essentially,  a  sophisticated 
version  of  weighted  scores  that  adds  statistics  and  levels 
of  details.  It  uses  a  three-dimensional  matrix  to  model 
the  selection  problem.  The  fourth  method,  Cost-Value, 
strives  to  reduce  the  subjectivity  found  in  weighted  score 
methods  and  to  associate  added  features  to  their  costs. 

It  is  especially  useful  to  equalize  differing  bid 
proposals. 

The  Interactive  Financial  Planning  System  (IFPS) 
is  interesting  in  that  it  is  a  computer  model  using 
natural  language  and  offers  several  gaming  facilities. 


The  particular  version  of  IFPS  reviewed  in  this  thesis  was 
tailored  for  use  in  analysis  of  government  acquisition  of 
large  computer  systems.  The  last  method/  Weighted 
Evaluation  of  Cost  Factors/  presents  a  simple  method  to 
incorporate  intangible  factors  into  cost  analysis. 

The  variety  of  methods  are  presented  in  more 
detail  in  the  following  subtopics.  The  author's  objective 
in  presenting  such  a  selection  is  to  provide  several 
techniques  currently  found  in  literature  that  are 
appropriate  for  use  during  the  vendor  selection  stage  in 
computer  acquisition.  The  manager  or  analyst  performing 
the  comparison  of  different  vendors'  proposals  can  select 
a  method  which  fits  his  analysis  needs  and  which  will  also 
be  acceptable  to  higher  management.  Some  of  the  methods 
presented  may  be  too  complicated/  require  too  much  timer 
or  may  not  conform  to  the  regulatory  environment.  The 
"best"  method  can  only  be  determined  by  the  individual (s) 
who  will  use  the  method. 

The  organization  of  the  rest  of  the  chapter  is 
such  that  the  reader/  only  desiring  an  overview  of  methods 
and  the  author's  evaluation  of  the  methods/  may  read  the 
introduction  to  the  chapter  and  the  comments  at  the  end  of 
each  detailed  subtopic.  The  reader  is  reminded  that  the 
full  text  on  each  subtopic  may  still  leave  detailed 
questions  unanswered/  since  the  purpose  of  this  chapter  is 


to  provide  the  essence  of  each  method  and  an  evaluation  of 
its  use.  More  details  of  the  methods  may  be  found  in  the 
sources  listed  in  the  bibliography  of  this  thesis. 


Cost/Benefit  Analysis 


Cost/Benefit  Analysis  (CBA) ,  or  Cost/Effectiveness 
Analysis,  can  be  found  in  many  sources  such  as  those 
included  in  this  thesis.  [1?  3;  4;  10;  25]  It  can  be 
considered  a  classic  and  could  be  used  at  many  other 
stages  than  just  vendor  selection. 


To  evaluate  the  acquisition,  installation,  or 
development  of  computer  equipment  and  software,  to 
measure  the  extent  of  any  improvement  in 
computer-based  systems,  or  to  determine  the  financial 
requirements  arising  from  planned  systems,  one  needs  a 
clear  picture  of  the  costs  involved.  To  further  judge 
the  efficiency  of  computer  hardware  and  software,  to 
determine  whether  a  change  in  a  system  is  indeed  an 
improvement,  or  to  justify  a  proposed  system  as  being 
worth  the  considerable  expenditures  envisaged,  one 
needs  to  define  and  measure  the  resulting  benefits. 

[3,  p. 61] 


CBA  usually  consists  of  five  steps,  which  may  vary 
slightly  from  author  to  author.  The  essence  of  the  steps 
is  that  all  costs  and  benefits  must  be  identified, 
classified,  recorded,  estimated,  and  analyzed  for  the 
useful  life  of  the  system.  Axelrod  adds  the  sixth  step  of 


decision  to  his  explanation:  {3,  p.86] 

1  Identification — What  are  the  various  costs  and 
benefits  that  pertain  to  a  specific  activity? 

2  Classification — How  should  the  various  costs  and 
benefits  be  categorized  to  facilitate  ensuing  steps? 

3  Recording —  What  are  the  actual  (or  estimated) 
magnitudes  of  the  cost  and  benefits? 

4  Analysis — How  should  the  various  cost  and  benefit 
figures  be  manipulated  to  represent  the  most 
significant  aspects  of  the  activity  in  terms  of  the 
decisions  to  be  made? 

5  Interpretation — What  is  the  meaning  of  the  results 
of  the  analysis? 

6  Decision — What  actions  should  result  from  a  correct 
interpretation  of  the  results  of  the  analysis? 

The  first  three  steps  of  identifying,  recording, 

and  classifying  costs  seem  to  have  two  approaches.  In  the 

approach  used  by  Axelrod,  the  costs  and  benefits  are 

classified  into  six  categories:  [3,  p.61] 

1  Tangible  vs.  intangible 

2  Direct  vs.  indirect 

3  Controllable  vs.  noncontrollable 

4  Fixed  vs.  variable 

5  Certain  vs.  uncertain 

6  One-time  vs.  ongoing 

Four  major  headings  are  defined:  Tangible,  direct; 
Tangible,  indirect;  Intangible,  direct;  and  Intangible, 
indirect;  within  these  major  headings  resource  types  are 
identified:  Equipment,  Software,  Supplies,  Staff,  and 
Space.  (See  Appendix  A.) 
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In  the  second  approach  to  classification,  Burch 
categorizes  cost  by  behavior,  function,  time,  and  type: 
(See  Fig.  3.1) 


Costs  by  Behavior 

1.  Variable  Costs.  These  costs  fluc¬ 
tuate  with  volume  changes  in  a  direct 
manner.  Examples  are  electrical 
power  and  supplies  (i.e..  if  the  volume 
of  work  increases,  the  use  of  electrical 
power  and  supplies  will  also  in¬ 
crease) 

2  Fixed  Costs.  These  costs  might 
vary  from  period  to  period,  but  this 
fluctuation  is  not  in  response  to  vol¬ 
ume  changes  in  a  particular  period. 
Examples  are  depreciation,  rent, 
taxes,  and  management  salaries 

Costs  by  Function 

t.  Development  Costs.  These  are 
costs  incurred  to  bring  something 
into  being  or  to  make  something  bet¬ 
ter,  more  useful,  etc, 

2.  Operational  Costs.  These  are 
costs  that  must  be  expended  to  make 
something  work  or  pertorm.  The  em¬ 
ployment  of  a  computer  operator  in¬ 
volves  operational  costs. 

Costs  by  Time 

1 .  Recurring  Costs  These  costs  are 
repeated  at  regular  intervals  Exam¬ 
ples  of  these  costs  are  payroll  and 
computer  rental  payments. 

2.  Nonrecurring  Costs.  These  are 
one-time  costs  or  costs  that  will  end 
at  some  specific  point  in  time.  The 
cost  of  computer  program  develop¬ 
ment  is  a  nonrecurring  cost.  (The  cost 
of  maintaining  computer  programs  is 
recurring  ) 

Costs  by  Type 

1.  Direct  Costs.  These  costs  repre¬ 
sent  expenditures  that  result  directly 
from  the  proposed  system. 

2.  Indirect  Costs.  These  are  over¬ 
head  costs  which  cannot  be  directly 
identified  with  the  elements  of  the 
proposed  system  and  are  apportioned 
among  various  areas  in  the  organiza¬ 
tion.  Examples  are  rent,  insurance, 
taxes,  management  salaries,  and  em¬ 
ployee  benefits. 

Fig.  3.1.  Burch's  variety  of  categories  for  cost 

[10,  p.419) 

The  major  headings  within  Burch's  Cost  and 
Effectiveness  Summaries,  Table  3.1,  are  by  resource  type 
with  the  different  types  of  costs  combined  under  the 
resource  heading,  although  Burch  admits  that  "management 
is  most  interested  normally  in  direct  costs."  Burch's 
headings  are  valuable  since  they  establish  the  fact  that 
the  actual  bid  costs  for  computer  configuration  are  just 
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one  part  of  the  total  cost  of  acquiring  a  system.  [10, 
pp. 421-2] 


TABLE  3  1 

COST  ITEMS  [10, ’pp. 422-4] 


1.  Computer  Configuration  Costs 

2.  Environment  Costs 

-  Power  requirements 

-  Air  conditioning 

-  Furniture  and  fixtures 

-  Miscellaneous  features 

3.  Physical  Installation 

4.  Training  Costs 

5.  Program  and  Program  Testing 

6.  Cost  of  Conversion 

7.  Cost  of  Operation 

-  Staff  costs 

-  Cost  of  supplies 

-  Equipment  maintenance 

-  Sytems  maintenance 

-  Power  and  light 

-  Insurance 

8.  Further  Systems  Work 


Burch  classifies  benefits  as  either  direct 
(tangible)  or  indirect  (intangible).  Direct  benefits  are 
those  cost  savings  "...  resulting  from  the  elimination  of 
an  operation,  or  from  the  increased  efficiency  of  some 
process."  An  example  of  a  direct  benefit  is  a  decrease  of 
$0.50  per  transaction  processed  by  a  new  system.  Indirect 
benefits  are  not  easy  to  quantify,  but  an  attempt  must  be 
made  to  identify  these  benefits  and  include  them  in  the 
analysis.  An  example  of  an  indirect  benefit  is  better 
customer  service.  Obviously,  both  types  of  benefits  must 
be  turned  into  a  dollar  value  to  be  included  in  the 


evaluation.  [10,  p.419-20] 

Whichever  method  of  classifying  is  chosen,  the 
recorded  costs  need  to  be  analyzed  for  valid  comparison. 
The  type  of  analyses  performed  and  the  information  that 
must  be  gathered  will  vary.  For  example,  lease  vs.  buy 
analysis  is  usually  performed  to  identify  the  best  option 
for  obtaining  the  different  vendors'  equipment.  To 
perform  this  analysis,  information  about  "...  lease 
charges,  purchase  price,  manitenance  schedule  and  charges, 
expected  useful  life,  estimated  resale  value,  and  specific 
tax  rules  and  regulations  must  be  gathered."  [3,  p.91] 

But  since  government  acquisition  directives  strongly 
recommend  purchasing  all  equipment,  this  type  of  analysis 
may  not  be  needed  for  a  government  study. 

Some  commonly  used  methods  of  analysis  are:  (1) 
Net  benefit;  (2)  Benefit-to-cost  ratio  (rate  of  return); 
(3)  Net  present  value  (NPV) ;  (4)  Relative  net  present 
value  (RNPV) ;  (5)  Internal  rate  of  return  (IRR);  and  (6) 
Payback  period.  [3,  p.93]  Appendix  A  contains  a  table 
which  briefly  describes  and  compares  these  methods  of 
analysis.  More  detailed  explanations  of  these  methods  may 
be  found  in  Axelrod's  Appendix  B  [3]  and  Burch's  Appendix 
A  [10]. 

After  the  results  of  the  analyses  are  obtained, 
they  have  to  be  interpreted.  This  step  may  involve 


performing  sensitivity  analysis  for  values  which  were  used 


that  have  a  high  degree  of  uncertainty.  In  simpler  words, 
suppose  a  computer's  estimated  resale  value  was  very  high 
for  one  vendor  and  caused  that  vendor  to  be  preferred; 
however,  the  high  estimate  was  without  a  good  basis, 
perhaps  just  a  guess.  Then  the  estimate  needs  to  be 
varied,  in  this  example  lowered,  and  the  resulting  effects 
briefed  when  the  CBA  is  interpreted.  The  analysis  methods 
themselves  have  assumptions  and  disadvantages  (see 
Appendix  B)  that  need  to  be  specified  with  the 
presentation  of  interpretations  of  the  CBA  to  management. 

The  decision  step  might  appear  to  be  quite  simple, 
with  CBA  presenting  the  most  economical  choice  of  computer 
system  However,  the  subjectivity  of  some  of  the 
estimated  numbers  combined  with  the  degree  of  uncertainty 
of  some  of  the  analysis  procedures,  usually  does  not  make 
the  CBA  answer  the  unquestionable  decision.  The  final 
decision  may  be  made  "...  based  on  overriding  intangible 
factors"  that  support  the  decision-makers  intuitive 
leaning.  [3,  pp.98-9] 

Comments 


The  Cost/Benefit  Analysis  method  should  be  studied 


because  it  is  used  so  frequently  and  its  accounting 
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procedure  methods  are  very  familiar  to  top  management.  A 
full  development  of  cost/benefit  analysis  is  beyond  the 
scope  of  this  thesis,  but  the  author  does  recommend  C.  W. 
Axelrod's  Computer  Productivity;  A  Planning  Guide  for  Cost 
Effective  Management  13]  for  an  in-depth  study  of  the 
topic;  or  Burch's  Information  Systems!  Theory  and  Practice 
[10,  pp. 418-24]  for  a  briefer  presentation  of  the  topic. 

A  definite  disadvantage  of  this  method  is  the 
level  of  expertise  needed  to  apply  and  interpret  the 
various  analyses  correctly.  For  this  reason,  it  may  be 
best  to  hire  experts  in  the  field  to  develop  the  CBA; 
even  with  expert  help,  the  in-house  staff  will  have  to 
gather  a  large  amount  of  data  and  provide  estimations. 

So,  another  disadvantage  is  the  time  and  effort  to  gather 
all  the  various  information  needed;  the  cost  of  preparing 
the  CBA  must  be  justified  by  the  importance  and  expense  of 
the  decision.  Certainly  major  system  procurements  would 
justify  a  CBA. 


Weiahi-ed  S.coie 

The  weighted  score  method  is  one  of  the  most 
common  methods  of  vendor  evaluation  to  be  found  in  current 
literature.  [10;  20;  22;  30;  33;  38]  It  is  also  one  of 
the  simplest  to  understand  and  perform.  It  may  be  used  to 
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evaluate  vendors  that  have  passed  all  the  mandatory 
requirements  for  the  new  system.  The  vendors  are  scored 
on  their  ability  to  provide  desired  system  features.  The 
features  are  each  weighted  to  reflect  the  relative 
importance  of  each.  The  vendor  with  the  highest  total 
weighted  score  is  considered  the  "best"  choice  and  is 
awarded  the  contract. 

As*  mentioned  before,  several  authors  have  written 
about  the  weighted  score  methodr  or  some  variation  by  a 
different  name;  but  Hussain  and  Hussain  presented  the 
clearest,  simplest  version.  A  portion  of  their  Table  €.5 
[20,  p. 110]  is  used  here  to  explain  the  weighted  score 
process,  as  applied  to  one  vendor.  The  full  table  may  be 
found  in  Appendix  B. 

The  list  of  desired  features,  or  decision  criteria 
in  column  1  of  Table  3.2,  should  already  have  been 
developed  before  this  stage  of  vendor  selection  has  been 
reached.  (Prior  listing  of  the  desired  features  ensures 
that  they  are  not  biased  toward  a  particular  vendor.) 
Weights  must  be  assigned  to  each  criterion  to  reflect  the 
relative  importance  of  each;  this  is  an  assigned  weight 
and  may  vary  with  different  acquisition  teams  and 
different  acquisitions.  In  the  sample  Table  3.2,  this 
team  thought  that  hardware  growth  needs  were  three  times 
as  important  as  real-time  capability. 


TABLE  3.2 

WORKSHEET  FOR  WEIGHTED  SCORE  METHOD  [20,  p.110] 


Vendor  A 


(1) 

(2) 

(3) 

(4) 

Wtd. 

Decision  Criterion 
Hardware 

Weight 

Score 

Score 

Meet  needs  of  growth 

3 

7 

21 

Throughput/S 

5 

8 

40 

Communications 

2 

4 

8 

Real-time  capability 

1 

1 

1 

Storage 

2 

8 

16 

I/O  interface 

2 

6 

12 

Site  restrictions 

1 

4 

4 

Reliability 

3 

9 

27 

Ease  of  use 

Total  for  hardware 

1 

6 

_£ 

135 

The  assignment  of  weights  may  not  be  a  simple 
process.  There  may  be  much  disagreement  among  the  team 
members  as  to  the  relative  importance  of  the  different 
criteria.  It  may  be  helpful  to  have  a  predetermined  plan 
of  how  to  reach  a  consensus  when  disagreement  arises,  i.e. 
who  has  veto  power  and  when/if  top  management  should  be 
called  in.  It  should  be  remembered  though  that  no  matter 
how  carefully  these  weights  are  assigned,  they  are 
subjective. 

Next,  the  vendors  are  scored  on  each  decision 
criteria,  column  3  of  Table  3.2.  Some  appropriate  scale 
needs  to  be  set  for  the  scoring;  the  example  in  Table  3.2 
used  a  scale  of  1-10,  with  10  as  high.  Vendor  A  rated 
relatively  high  on  reliability,  but  very  low  on  real-time 
capability.  Scoring  is  another  activity  that  is 


subjective.  Subjectivity  can  be  reduced  by  having  experts 
in  the  particular  areas  score  them.  It  is  also  best  if 
the  weight  assigned  to  the  item  is  not  known  by  the 
scorer,  so  the  scorer  is  ”...  not  prejudiced  by  the 
relative  importance"  of  the  item.  [30,  p.52]  ”...  Fair 

scoring  may  require  considerable  effort,  involving 
literature  searches,  calculations,  and  customer 
satisfaction  checks.”  [20,  p.lll] 

The  weighted  scores,  column  4  of  Table  3.2,  are 
calculated  by  multiplying  each  of  the  weights,  column  2, 
by  the  vendor's  score,  column  3.  The  vendor's  total 
weighted  score  is  merely  the  sum  of  the  individual 
weighted  scores.  After  all  vendors'  total  weighted  scores 
have  been  calculated  for  all  areas,  the  vendor  with  the 
highest  accumulation  of  points  is  the  winner. 

The  weighted  score  method  also  appears  as  an 
integral  part  of  more  complicated  methods,  like  the 
ELECTRE  [33],  an  on-line  decision-aid  program  based  on 
multicriteria  decision  theory.  ELECTRE  facilitates 
sensitivity  analysis  and  warns  the  decision  maker  when  one 
alternative  may  contain  important  factors  which  are  rated 
too  low  to  be  a  good  overall  choice.  For  example,  if 
vendor  stability  was  rated  very  low,  the  choice  of  that 
vendor  could  result  in  owning  equipment  with  no  company  to 
back  it. 
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Comments 


Weighted  score  method  is  probably  most 
appropriately  used  as  a  sole  decision  method  for  smaller 
acquisitions.  It  is  also  fine  for  large  acquisitions  with 
choices  between  almost  identical  costs  and  features.  The 
major  weakness  of  this  method  is  its  subjectivity, 

i 

although  some  steps  can  be  taken  to  reduce  subjectivity. 

Another  weakness  is  the  inability  to  relate  the  ratings  to 
costs.  Even  with  the  weaknesses  there  are  still  some 
features  that  are  better  rated  than  "costed",  i.e.  vendor 
stability.  The  ease  of  using  and  understanding  this 
method  may  make  it  the  best  choice. 

fayoif  Matri* 

The  Payoff  Matrix  is  presented  by  Martin  as  both  a 
model  which  can  be  used  as  a  "quantitative  tool”  and  as  a 
checklist  for  decision  making.  [29,  p.18]  Be  lists  the 
elements  necessary  to  build  a  payoff  matrix:  (1) 
alternatives,  Ai;  (2)  states  of  the  future,  Si;  (3) 

probabilities  of  the  states  occurring,  Prob(Si);  and  (4)  ] 

payoffs  for  each  combination  of  alternative  and  state  of 
the  future,  Outcome  Ai,Si.  [29,  p.15]  The  payoff  matrix 


takes  the  general  form  shown  in  Table  3.3. 

TABLE  3.3 

THE  PAYOFF  MATRIX  [29,  p.16] 

Alternatives  States  of  the  Future 

SI  S2  S2 

Prob(Sl)  Prob (S2)  Prob(S3) 

Al  Outcome  1,1  Outcome  1,2  Outcome  1,3 

A2  Outcome  2,1  Outcome  2,2  Outcome  2,3 

A3  Outcome  3,1  Outcome  3,2  Outcome  3,3 

The  generation  of  the  alternatives  can  be 
difficult.  Usually,  the  alternatives  can  best  be 
generated  by  a  team  with  a  wide  perspective  and  sufficient 
time  for  "incubation  of  ideas.”  In  the  specific  example 
of  choosing  a  computer,  the  different  vendors  being 
considered  constitute  the  alternatives  for  the  model.  [29, 
pp.15-6] 

The  determination  of  states  of  the  future  first 
requires  that  the  planning  horizon  be  specified.  The 
longer  the  time  span,  the  more  difficult  it  will  be  to 
predict  the  states  and  their  probabilities.  Each  state's 
probability  is  a  quantification  of  the  possibility  of  that 
state  occurring  in  the  future.  The  probabilities  can  be 
statistically  computed  from  a  ranking  of  relative 
likelihoods.  Martin  recommends  that  a  heterogeneous  group 
with  "...  a  broad  perspective  and  with  time  allowance  for 
incubation  of  ideas"  be  used  to  determine  states  and  their 


relative  probabilities.  [29,  p.16] 

Next,  payoff  values  are  determined  for  each 
combination  of  alternatives  and  states.  The  various 
factors  affecting  the  selection  process  are  identified  and 
ranked  in  order  of  relative  importance.  The  ranks  can 
again  be  statistically  translated  into  weights.  If  group 
consensus  over  the  relative  importance  of  the  factors  can 
not  be  reached,  Martin  suggests  the  use  of  some  plan  like 
considering  which  members  input  should  receive  the  most 
importance  (i.e.,  "highest  ranking  member").  [29,  p. 16-17] 
Table  3.4  displays  possible  selection  factors  for 
the  task  of  selecting  one  brand  of  computer  from  several 
alternatives.  The  factors  have  been  ranked  and  their 
corresponding  weights  calculated. 


TABLE  3.4 

CONVERSION  OP  RANKINGS  TO 
NUMERICAL  WEIGHTS  [29,  p.16] 


Factor 

Rank 

Weight 

Hardware 

3 

0.20 

Software 

4 

0.13 

Skill/Training 

1 

0.33 

Vendor  Services 

2 

0.27 

Management  Acceptance 

_5_ 

-  JM?J_ 

n  *  5  factors 

k  =  15 

1.00 

Weight  ■  (n  -  Rank  +  1)  /  k 


Table  3.5  shows  how  the  primary  factor ,  Vendor 


Services,  was  further  subdivided,  and  weights  were 
calculated  for  each  subfactor.  This  process  allows  the 
dissection  of  factors  down  to  whatever  level  of  detail 
desired. 


TABLE  3.5 

SUBFACTOR  WEIGHTS  FOR 
VENDOR  SERVICES  [29,  p.17] 

Subfactor  Subfactor  Weight 


Education  0.07 
Maintenance  0.11 
Systems  Design  0.04 
Trouble  Shooting  0.05 


Total  0.27 


After  the  weighted  factors  have  been  derived,  the 
values  must  be  summarized  for  each  combination  of 
alternative  and  state.  To  again  return  to  our  computer 
selection  example,  each  brand  is  ranked  for  each 
subfactor;  the  ranks  will  probably  vary  for  each  brand  as 
different  states  are  considered.  The  individual  ranks  are 
multiplied  by  the  weights  for  the  subfactors;  then  summed 
to  produce  the  Factor  Weighted  Total,  see  Figure  3.2.  The 
various  weighted  totals  for  the  primary  factors  are  summed 
to  find  the  Grand  Weighted  Total.  (Note:  value  may  be 
negative  when  objective  is  to  minimize  costs.)  [29,  p.16] 
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EXHIBIT  5 

CONVERTING  SUBFACTOR 

VALUES  INTO  GRAND  WEIGHTS 

State  of  Nature  1 

Computer  C  I 

Weighted 

Subfactor 

Weight 

Rank  Rank 

Education 

0.07 

1  0.07 

Maintenance 

0.11 

3  0.33 

Systems  Design 

0.04 

3  0.12 

Trouble  Shooting 

0.05 

2  0.11 

Total 

0.27 

N/A 0.63  v 

Factor  Weighted  \  1 

Total  \ 

Weighted  \ 

Factor 

Rank  . 

Hardware 

0.53  / 

Software 

0.29  / 

Skill/Training 

0.74  J 

Vendor  Services 

0.63  “ 

1  Management  Acceptance 

0.11 

Total 

*  2.30 

|  Grand  Weighted  Total'  j 

Fig.  3.2.  Matrix  model-  converting 
subfactors  to  grand 
weights  [29,  p.17] 


v- 
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The  process  just  described  is  graphically  depicted 
in  Pig.  3.3. 


After  all  the  Grand  Weights  have  been  calculated, 
the  matrix  can  be  solved  to  arrive  at  expected  values  for 
each  alternative.  Expected  values  are  calculated  by 
multiplying  each  state's  probability  by  its  grand  weight 
and  summing  the  results  for  each  state.  The  computation 
for  computer  C  of  Figure  3.4  is: 


(0.70  x  2.30)  +  (0.30  x  1.21)  =  1.97 


QUANTITATIVE  SOLUTION 
TO  THE  PAYOFF  MATRIX 


Computer 

States  of  the  Future 

Expected 

Value 

1 

2 

p  =  0.70 

p  =  0.30 

A 

2.05 

2.63 

2.22 

B 

1.52 

2.00 

*1.66 

C 

2.30 

1.21 

D 

3.15 

3.00 

7  3.11 

Select  / 
This  / 
Alternative 


Pig.  3.4.  Solution  of  payoff  matrix 
129,  p.18] 

Martin  suggests  that  the  matrix  may  not  really 
need  to  be  solved  to  make  an  effective  decision.  The 
systematic  framework  of  the  matrix  that  has  forced 
enumeration  of  "...  all  important  factors  for  evaluating 
our  alternatives  and  to  assess  the  relative  importance  of 
these  factors"  may  have  already  provided  the  input 
necessary  to  make  an  effective  decision.  He  thinks  that 
the  payoff  matrix  model  "...  is  far  more  valuable  as  a 
checklist  for  decision  making",  see  Appendix  C.  [29,  p.18] 
Martin  warned  "quantitive  addicts"  that  if  the 
matrix  solution  is  used,  to  remember  that  it  relies 
entirely  on  "...  the  weights  assigned  to  selection  factors 
and  the  probabilities  given  to  the  states."  [29,  p.18]  He 
recommended  doing  sensitivity  analysis  to  discover  the 
most  important  factors  contributing  to  the  ultimate 


decision.  This  is  done  by  varying  some  of  the  weights  and 
probabilities  to  see  if  the  decisions  are  changed. 

Factors  identified  as  critical  can  then  be  further 
researched  to  assure  that  the  correct  weights  and 
probabilites  have  been  used.  [29 r  p.18] 

Comments 


This  author  re-emphasizes  that  the  greatest 
weakness  of  this  method  is  subjectivity.  Sensitivity 
analysis  should  help,  but  unless  this  model  is  on  the 
computer,  recomputing  the  values  multiple  times  will  get 
tiresome  and  therefore  error-ridden.  The  checklists, 
multi-dimensional  matrices,  and  the  neatness  of  the 
"statistically  derived  solutions"  adds  a  lot  of  flashiness 
to  what  is  basically  a  weighted  score  method.  If  weighted 
score  was  the  best  choice  for  comparing  very  similar  bids, 
but  the  boss  wants  something  more  sophisticated,  then  the 
Payoff  Matrix  is  a  good  choice. 

Cost-Value 

Hussain  and  Hussain  present  the  cost-value  method 
as  an  "...  attempt  to  equalize  bids  of  features  so  that 


costs  can  be  compared.  Costs  of  desired  features  not 


included  in  proposals  are  added  to  each  vendor's  bid." 

120/  p.112]  The  cost  of  each  additional  feature  is  added 
to  the  orginal  bid  when  a  vendor's  proposal  did  not 
include  that  feature  or  subtracted  from  the  original  total 
when  a  requirement  is  exceeded.  Each  additional  feature 
should  also  have  an  estimate  developed  for  the  option  of 
doing  without  that  feature;  this  value  should  be  added  to 
the  bid  if  it  is  less  than  acquiring  the  feature. 

For  example,  if  a  certain  software  feature  is 
desired  but  was  not  included  in  the  vendor's  original  bid, 
the  life  cycle  cost  of  the  software  should  be  estimated 
for  purchase,  in-house  development,  and  doing  without  the 
software.  If  the  purchase  of  the  software  plus  its 
maintenance  was  $19,000,  in-house  development  and 
maintenance  was  estimated  at  $16,000,  but  the  cost  of 
degraded  service  from  doing  without  the  software  was  only 
$12,000,  then  the  correct  decision,  based  on  cost,  would 
be  to  do  without  the  software.  Therefore,  the  $12,000 
would  be  the  correct  estimate  to  include  in  the  cost-value 
analysis  for  this  added  feature.  [20,  pp. 112-3] 

An  example  of  a  value  that  could  be  subtracted  for 
the  overfulfillment  of  a  requirement  is  promised  delivery. 
Table  3.6.  If  the  equipment  is  needed  by  June  1987  and 
Vendor  C  can  deliver  three  months  early,  then  Vendor  C  is 
given  credit  for  exceeding  thr  requirement  by  estimating 
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the  savings  that  will  result  from  the  early  delivery.  If 
the  savings  is  estimated  at  $2000 ,  then  $2000  is 
subtracted  from  Vendor  C's  original  bid.  Likewise,  if 
Vendor  B  can't  make  the  date  required  and  will  be  three 
months  late,  costing  $4000,  then  Vendor  B  will  have  $4000 
added  to  their  original  bid.  Note  that  Vendor  A's 
adjustment  is  zero  for  on-time  delivery. 

TABLE  3.6 

VALUE  TEMPLATE  FOR  DELIVERY  DATES  [20,  p.113] 

Vendor  Vendor  Vendor 

ABC 

Date  of  delivery  June  87  Sept  87  March  87 
Value  0  +$4,000  -$2,000 

Hussain  and  Hussain  included  an  actual  cost-value 
application  for  a  DBMS  acquisition;  the  case  was  chosen 
because  the  cost-value  method  works  equally  well  for 
software,  although  usually  applied  to  hardware 
acquisitions.  The  costs  shown  are  real,  but  are 
consolidated  into  less  features  and  applied  to  only  two 
vendors.  Table  3.7  shows  that  Vendor  B's  proposal  price 
was  half  of  Vendor  A’s,  but  after  the  value  of  omitted 
features  were  added,  Vendor  A's  equivalent  price  was  much 
lower.  One  of  the  high  costs,  conversion,  must  always  be 
considered  when  acquiring  new  hardware  too.  'The  most 
efficient  hardware  may  turn  out  to  be  unacceptable  because 


of  such  conversion  costs.”  [20,  p.114] 


TABLE  3.7 

COST  CALCULATIONS  FOP  SELECTION  OF  DBMS  [20,  p.114] 


Vendor  A 

Vendor  B 

Cost  Items 

($000) 

($000) 

Cost  of  vendor  proposal 

$208 

$102 

Interface  to  a  higher  level 
language;  i.e.  BASIC 

42 

18 

Natural  language  query  facility 
with  communications  interface 

20 

50 

Equipment  interdependence 
( 52 k/ma chine) 

51 

104 

Data  element  dictionary 

— 

57 

Supporting  equipment  necessary 

— 

20 

Inverted  file 

— 

70 

Recovery  procedures 

— 

55 

Security 

55.5 

5 

Conversion  of  data  base 

_ 34 

-  3-5. 

Total 

$410.5 

$596 

The  cost-value  method  reduces  the  subjectivity  of 
the  vendors'  comparisons,  a  problem  with  scores  and 
weighted  value  judgments.  Some  problems  with  subjectivity 
still  remain  ”...  when  estimating  the  life  of  a  system, 
the  cost  of  degraded  service  due  to  lack  of  features,  or 
the  benefits  to  be  gained  from  overfulfilled 
requirements.”  [20,  p.114]  Subjectivity  could  be  a 
greater  problem  if  the  cost  figures  are  not  carefully  and 
objectively  developed  by  experts. 

A  major  disadvantage,  discussed  by  the  Bussains, 
is  the  time  and  effort  required  to  complete  the 
evaluation.  The  DBMS  case,  simplified  in  Table  3.7, 


actually  took  a  seven-person  team  three  and  a  half 
man-years  to  complete.  This  was  a  very  experienced, 
technical  team  consisting  of  a  head  of  information 
systems,  a  consultant,  three  systems  programmers,  and  two 
applications  programmers.  [20,  p.114] 

Comments 


This  author  feels  that  the  cost-value  method 
offers  a  good  structured  approach  for  comparing  dissimilar 
proposals,  where  the  basic  cost  could  be  deceiving  or 
hidden  by  additional  features  that  are  difficult  to 
compare.  The  idea  of  relating  value  to  cost  and  making  a 
more  objective  comparison  is  good;  also  managers  like  to 
see  dollar  figures.  The  inclusion  of  "nice  to  have 
someday"  features  into  the  basic  bid  was  an  approach  not 
found  in  other  authors'  work.  Furthermore,  the  Hussain 
and  Hussain  text  is  a  good  reference  for  the  novice  to 
information  resource  management. 

Interactive  Financial  Planning  System 

The  Interactive  Financial  Planning  System  (IFPS) 


was  selected  by  LaRue  and  DeLorenzo  for  developing  a  model 
to  study  government  ADP  acquisition.  [24]  IFPS  is 


marketed  by  Executive  Systems  Corporation,  Austin,  Texas. 
The  advantages  of  IFPS  are: 

IFPS  provides  for  natural  language  development  of 
financial  planning  models.  In  addition  to  built-in 
financial,  statistical,  and  mathematical  functions, 
IFPS  is  capable  of  performing  interactive  what  if, 
sensitivity,  and  goal  seeking  analyses.  Additionally, 
IFPS  is  capable  of  performing  risk  analysis  through 
simulation  (Monte  Carlo  analysis).  The  extensive, 
natural  language  modeling  capabilities  and  the 
flexibility  of  IFPS  combined  with  its  availability  for 
a  number  of  different  manufacturer's  equipment,  made 
IFPS  a  natural  selection  for  development  of  the 
composite  model.  [24,  p.60-1] 


The  cost  model  developed  by  LaRue  and  DeLorenzo 
using  IFPS  was  based  on  three  large  system  acquisitions  by 
the  Navy.  The  output  from  their  model  matches  the 
decision  milestones  used  during  the  various  stages  of  the 
Navy's  life-cycle  acquisition  process  and  can  "...  clearly 
communicate  the  real  cost  of  a  program  to  decision  makers 
at  each  milestone."  [24,  p.76] 

Identifying  cost  drivers  is  a  critical  part  of  any 
good  economic  analysis.  Since  the  cost  elements  included 
in  the  model  were  extracted  from  the  records  of  real 
acquisitions,  the  costs  identified  are  not  just  good 
guesses,  but  have  been  verified  as  real.  These  real  costs 
have  been  integrated  with  costs  that  will  arise  from 
regulatory  and  organizational  requirements,  costs  which 
"...  are  seldom,  if  ever,  accounted  for  in  determining  the 
cost  of  a  project."  [24,  p.76J 
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The  cost  elements  are  grouped  into  the  following 
major  "modules":  personnel;  contractor  support;  other 
agency  or  department,  material  and  equipment,  and  other 
direct  costs.  [24,  p.127]  The  major  modules  are  broken 
down  into  major  categories;  for  example,  personnel  costs 
contains  major  categories  of: 

-  Project  Office  Personnel 

-  Organizational  Support  Personnel 

-  Support  Activity  Personnel 

-  Executive  Review  Personnel 

-  Approval  Level  Personnel 

-  Acquisition  Processing  Personnel  [24,  p.78] 

The  major  categories  are  further  broken  down  to  cost 
element  structures;  within  personnel  five  levels  of 
personnel  costs,  ranging  from  clerical  to  senior 
executive,  may  be  used. 

A  final  good  feature  included  in  the  model  is  the 
computation  of  lost  opportunity  costs.  "Opportunity  costs 
were  considered  to  be  the  projected  yearly  cost  savings 
attributed  to  the  new  system."  [24,  p.78]  Lost 
opportunity  costs  were  discounted  back  to  the  fourth  year 
in  this  eight  year  model  and  added  to  the  adjusted  total 
project  cost.  This  adjustment  was  provided  to  "...  more 
accurately  (represent)  the  true  costs  of  the  system."  [24, 
p.78] 


moments 
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The  writer  of  this  thesis  has  provided  a  very 
short  review  of  a  detailed,  well-documented,  excellent 
piece  of  work.  LaRue  and  DeLorenzo's  model  looks  like  one 
of  the  best  detailed,  computer  models  available.  It  is 
especially  valuable  in  that  it  incorporates  governmental 
regulations  and  military  organizational  guidelines  into 
the  model.  The  on-line  gaming  capability  of  IFPS  also 
seems  like  a  very  promising  decision-making  tool.  The 
cost  of  using  this  model  (gathering  the  information  and 
inputting  it,  making  decisions,  and  computer  time  to  run 
the  model)  will  be  substantial;  therefore,  this  would 
only  be  warranted  for  a  large  acquisition  project. 


Weighted  Evaluation  of 


Vaid-Raizada's  presentation  of  Weighted  Evaluation 
of  Cost  Factors  combines  monetary  annual  costs  with 
weighted  ratings  of  intangible  factors  into  one  value, 
which  reveals  the  best  system.  The  emphasis  of 
Vaid-Raizada's  work  is  on  the  identification  and 
evaluation  of  intangible  cost  factors,  which  the  following 
chapter  discusses  further.  He  has  chosen  this  emphasis 
because  he  feels  that  intangible  factors  are  often  ignored 
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or  not  given  due  importance,  often  resulting  in  the 
selection  of  the  wrong  computer.  [40,  p.30] 

The  weighted  evaluation  method  suggests  evaluating 
the  intangible  factors  after  the  selection  process  has 
narrowed  down  to  a  few  available  systems.  A  simple, 
seven-step  approach  is  presented  to  develop  the  total 
weighted  intangible  rating  for  each  possible  choice:  (See 
Table  3.8) 


TABLE  3.8 


INTANGIBLE 

RATING 

EVALUATION 

FOR  ABC 

SYSTEM 

[40,  p. 

35) 

1 

2 

3 

4 

5 

Ease  of  use 

100 

16.67 

90 

15.00 

Vendor  support 

90 

15.00 

95 

14.25 

Maintenance  service 

80 

13.33 

70 

9.33 

Hardware  char. 

80 

13.33 

50 

6.66 

Software  char. 

70 

11.67 

75 

8.75 

Systems  performance 

65 

10.83 

80 

8.66 

Reliability 

60 

10.00 

60 

6.00 

Security 

55 

9.17 

70 

6.42 

Total 

600 

100. 

— 

75.07 

STEP  1.  Make  a  list  of  all  the  intangible  factors. 
STEP  2.  Determine  the  relative  importance  of  each  and 
list  factors  from  most  to  least  importance,  column  1. 
STEP  3.  Assign  an  importance  value  to  each,  using  100 
as  maximum,  column  2. 

STEP  4.  Adjust  the  importance  value  (to  sum)  to  100, 
column  3  figures.  (Values  in  column  2  divided  by 
total  of  column  2.) 

STEP  5.  Evaluate  each  intangible  factor  for  each 
alternative  system.  (Other  alternatives  not  shown) 
STEP  6.  Determine  the  individual  weighted  evaluation 
ratings,  column  5.  (Multiply  column  3  by  column  4.) 
STEP  7.  Total  weighted  evaluation  for  each 
alternative.  (Sum  column  5.) 


Next  the  costs  of  each  system  are  analyzed 
Vaid-Raizada  does  not  present  how  initial  costs  of 


hardware,  software/  or  installation  are  derived;  nor  how 
to  determine  annual  operating  and  maintenance  costs.  His 
final  annual  capitol  costs  are  derived  using  an  eight-year 
life  cycle  and  discount  rate  of  10  percent.  The  total 
annual  cost  is  the  sum  of  the  annual  capitol  costs  and  the 
annual  operating  and  maintenance  costs  for  each 
alternative  system.  [40,  p.36] 

The  final  step  is  to  incorporate  the  intangible 
rating  into  the  annual  cost  figure,  arriving  at  the 
weighted  evaluation  figure  for  each  alternative.  Table 
3.9  shows  an  example  comparing  three  systems,  where  annual 
cost  alone  would  lead  to  the  selection  of  system  ABC. 
However,  when  the  total  annual  costs  (row  1)  are  divided 
by  the  total  weighted  intangible  ratings  (row  2),  the 
results  (row  3)  reveal  system  XYZ  as  the  best  choice.  [40, 
p.36] 


TABLE  3.9 

WEIGHTED  EVALUATION  OF  COST  FACTORS  [40,  p.36] 

SYSTEM  ABC  PQR  XYZ 

1.  Total  annual  cost  $34,601  $37,307  $39,556 

2.  Total  weighted  0.7507  0.8054  0.9149 

intangible  rating 

3.  Weighted  evaluation  $46,092  $46,321  $43,235 

of  cost  factors 
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Comments  i 

i 

This  approach  is  admirable  in  that  it  stresses  the  j 

i 

importance  of  intangible  factors  and  provides  a  systematic  ; 

method  to  develop  those  factors  and  incorporate  them  into  i 

t 

the  decision  process.  The  non-specification  of  the 
derivation  of  the  total  costs  is  no  big  problem  as  it 
leaves  the  analyst  free  to  choose  his  favorite  approach. 

This  author  is  concerned  that  no  mention  was  made  that  the  | 

subjectivity  of  the  total  weighted  intangible  costs  could 
lead  to  acceptance  of  the  "best  buy"  that  really  wasn't 
the  best.  (Varying  ABC's  intangible  rating  by  +0.05  makes 
it  the  best  buyl) 

A  similar  approach,  called  the  Brocato  Method,  was 
presented  by  McMillan.  [30,  part  IV,  pp.51-6]  The 
weighted  score  portion  of  this  method  was  basically 
concerned  with  technical  rating  instead  of  intangible 
factors.  (See  Brocato 's  list  in  Appendix  D. )  In  this 
method,  the  total  rating  points  are  divided  by  the  annual 

i 

cost  to  find  a  "value  per  dollar  figure".  [30,  p.52] 

Both  of  these  methods  strive  to  combine  cost 
factors  with  those  factors  which  seem  to  be  measured  best 
by  some  type  of  rating.  This  seems  to  be  a  good  approach, 

» 

but  care  must  be  taken  when  giving  the  subjective  rating  | 

such  an  impact  on  the  decision.  j 

< 

« 

f 

t 


CHAPTER  IV 


ESTIMATION  OF  INTANGIBLE  COSTS  AND  BENEFITS 

What  is  an  intangible  cost  or  benefit?  Axelrod 
identifies  them  as  the  converse  of  tangible  costs  and 
benefits,  "...  which  can  be  readily  identified  and 
measured."  He  further  elaborates  that  intangible  costs 
and  benefits  "...  may  be  easy  to  identify  but  difficult  to 
measure,  such  as  the  cost  of  delay  caused  by  a  breakdown 
in  equipment  ...  or  may  be  difficult  to  even  identify, 
such  as  the  improvement  in  corporate  image.”  [3,  p.63] 
Intangible  costs  and  benefits,  or  intangible  factors,  are 
very  difficult  to  deal  with,  so  too  often  they  are  simply 
omitted  from  analysis. 

Experts  agree  that  ignoring  or  improperly  treating 
intangible  factors  can  result  in  the  possible  selection  of 
improper  equipment  or  overall  bad  project  decisions.  [3; 
25;  28;  30;  37;  40]  Vaid-Raizada  reports  that  possibly 
half  of  all  installed  computer  systems  are  selected 
improperly  due  to  inadequate  attention  paid  to  intangible 
factors.  [40,  p.30]  Lay  cited  an  example  of  a  two-year, 
1.7  million  dollar  scrapped  project,  that  even  if 
finished,  never  accounted  for  the  intangible  factor  that 


user  expertise  was  inadequate  to  ever  run  the  system!  [25, 
p. 30]  Unfortunately,  such  horror  stories  are  common. 

Most  systems  people  either  lived  through  such  a  story  or 
have  heard  about  system  changes  that  grew  out-of-budget 
and  out-of-time  due  to  intangible  factors  never  considered 
during  planning  phases.  It  is  easy  to  find  examples  of 
the  importance  of  including  intangible  factors  in  the 
decision  process,  but  difficult  to  find  just  how  to  do  it. 

While  experts  have  warned  that  ignoring  intangible 
factors  is  a  common  and  often  disastrous  practice,  little 
concrete  guidance  can  be  found  as  to  the  correct  treatment 
of  intangibles  during  the  computer  selection  process. 

This  chapter  incorporates  what  guidance  was  available  with 
some  of  this  author's  thoughts  on  the  subject.  These 
thoughts  are  further  developed  in  the  following  chapter, 
using  data  from  the  case  study. 


Most  complex  undertakings,  and  selection  of  a 
large  computer  system  would  seem  to  qualify  as  complex, 
seem  to  benefit  from  some  systematized  approach.  A  very 
common  form  of  systematizing  found  in  the  government  is 
the  checklist.  There  are  checklists  covering  everything 
from  how  to  operate  a  piece  of  machinery,  to  prepare 


documents  for  system  changes,  or  to  answer  a  threatening 
phone  call.  The  checklist  leads  new  troops  through 
intricate  steps,  serves  as  a  memory  jog  for  experienced 
people,  and  provides  steps  for  managers  to  use  for  quality 
control  and  to  chart  progress. 

This  systematic  checklist  approach  seems  a 
plausible  way  of  handling  the  inclusion  of  intangible 
factors  in  the  selection  process.  There  are  different 
types  of  checklists  that  would  seem  appropriate  for 
treatment  of  intangibles.  One  might  be  an  overall  plan  of 
how  to  treat  intangibles.  Another  could  be  a  listing  of 
the  steps  that  should  be  taken  to  capture  information  on 
intangible  factors  during  each  step  of  the  selection 
process.  Still  another  checklist  might  enumerate  all  the 
possible  intangible  factors,  grouped  into  stages  or 
categories,  that  have  been  observed  in  the  past  for  large 
computer  acquisitions.  A  final  checklist  might  outline 
the  evaluation  process  and  describe  the  method  for 
inclusion  of  intangible  factors  into  the  final  selection 
process. 

The  first  checklist  to  develop  is  an  overall  plan 
of  how  to  treat  intangibles.  If  your  company's  approach 
is  to  ignore  them  because  they  can't  be  quantified,  then 
your  checklist  will  be  easy  to  develop.  Otherwise,  your 


selection  committee  will  need  to  clarify  the  following: 

(1)  What  is  an  intangible? 

(2)  How  will  intangible  factors  be  reported? 

(3)  Who  will  evaluate  them? 

(4)  How  do  they  fit  into  the  selection  method  to  be 
used? 

(5)  What  level  of  importance  should  they  receive? 

(As  important  as  real  money  factor?) 

(6)  Will  sensitivity  analysis  be  performed? 

(7)  How  will  we  report  this  to  management? 

This  list  of  questions  should  be  addressed  at  the 
beginning  of  the  selection  process;  there  may  be  more 
questions,  and  possibly  answers,  your  selection  committee 
can  add  during  a  brainstorming  meeting  on  intangibles. 

The  process  of  looking  at  the  inclusion  of  intangible 
factors  is  important  even  if  the  final  decision  is  to  not 
address  intangibles  due  to  the  difficulty  to  evaluate 
them.  At  the  very  least,  the  decision  makers  will  be  made 
aware  of  the  numerous  intangible  factors  that  could  affect 
what  appears  to  be  the  "best"  dollar  choice. 

If  the  decision  is  made  to  include  intangible 
factors,  then  the  second  question  of  how  to  report 
intangibles  needs  to  be  decided  before  vendors'  proposals 
are  solicited.  The  same  format  for  reporting  and 
evaluating  tangible  factors  may  work  with  modifications  or 
a  new  checklist  procedure,  complete  with  new  forms  may 
need  to  be  developed.  (Chapter  5  contains  a  form  for 
capturing  information  on  intangible  factors.) 


Ideally  the  selection  committee  should  compile  a 
master  list  of  all  the  intangible  factors  to  be  included 
in  the  evaluation  process.  An  example  master  list  of  all 
possible  intangible  factors  found  during  research  would  be 


an  invaluable  aid;  unfortunately,  such  lists  do  not  seem 
to  exist.  If  this  is  a  mature  computer  system,  perhaps 
this  is  not  the  first  replacement  of  the  system  and  past 
experience  will  provide  some  ideas.  (Chapter  5  provides  a 
master  list  from  the  case  study  that  may  provide  some 
suitable  starting  point.) 

After  the  intangible  factors  have  been  identified, 
then  some  appropriate  measures  can  be  taken  to  include 
intangibles  in  the  selection  decision.  The  way 
intangibles  are  included  would  vary  with  the  type  of 
selection  method  being  used.  The  previously  reviewed 
methods  in  Chapter  3  ranged  from  Cost/Benefit  Analysis, 
where  all  intangible  factors  would  have  to  be  translated 
into  dollars;  to  rating  methods  like  Weighted  Score, 
where  intangibles  need  only  to  be  rated  for  each  vendor. 
The  conversion  of  intangibles  to  dollar  figures  seems 
risky  at  best,  since  intangibles  by  definition  defy 
quantifying.  Whatever  form  the  intangible  takes,  it  will 
only  be  an  estimation,  but  not  all  estimations  are  equally 
good. 


One  of  the  biggest  problems  of  including 
intangible  factors  in  the  selection  process  is  the 
subjectivity  of  the  values  assigned  to  those  factors.  By 
definition,  intangibles  will  always  be  difficult  to 
quantify;  it  doesn't  matter  whether  the  value  assigned  is 
in  dollars  or  a  rating,  that  value  will  have  to  be 
subjective.  The  goal  is  to  restrict  the  degree  of 
subjectivity  as  much  as  possible.  Additional  goals  are  to 
establish  clearly  the  method  of  estimation,  identify  any 
important  assumptions,  and  indicate  the  degree  of 
confidence  the  estimator  has  in  the  estimate. 

One  obvious  way  to  reduce  subjectivity  is  to  gain 
the  information  for  evaluation  from  sources  which  have  no 
bias  as  to  the  decision.  For  example,  when  questioning 
the  vendor  on  user  friendliness  of  their  products  the  best 
possible  picture  will  be  painted;  however,  studying 
impartial  user  polls  from  reputable  sources  will  provide  a 
less  biased  picture.  Another  possible  source  of  bias 
could  be  using  an  in-house  evaluator  with  a  definite 
preference  for  a  certain  product.  This  person's  source  of 
bias  could  be  simple  resistance  to  change  or  desire  to 
provide  "what  the  boss  really  wants  to  hear". 

Another  simple  way  to  reduce  subjectivity  is  to 


parcel  out  the  various  areas  to  be  evaluated  to  experts  in 
those  fields.  The  evaluation  should  be  made  without 
information  on  the  relative  importance  of  the  factor  to 
the  decision  process  (weight),  so  the  evaluator  would  not 
be  influenced  by  the  importance  of  the  factor.  Evaluators 
could  also  be  asked  to  provide  best,  average,  and  worst 
estimates  along  with  the  probability  that  each  will  occur. 
Such  information  can  be  used  to  calculate  expected  values, 
which  are  better  averages  to  use  for  estimation. 

Several  expert  estimates  could  be  averaged.  With 
this  technique  it  would  be  best  to  ask  each  expert  to 
develop  their  estimates  independently,  so  they  will  not 
unduly  influence  individual  decisions.  [10,  p.421]  Each 
expert  should  provide  the  basis  for  their  estimation,  if 
it  was  just  based  on  experience  it  should  be  so  identified 
versus  hours  of  reviewing  current  literature.  The  person 
responsible  for  combining  the  estimates  may  desire  to 
weight  some  estimates  before  averaging. 

Identifying  the  probability  that  the  estimation  is 
correct  is  important  no  matter  how  the  estimation  was 
derived.  Identifying  the  critical  assumptions  is  also 
essential.  If  the  assumptions  should  change  during  the 
selection  process,  then  the  whole  decision  may  be  changed. 
Intangible  factors  that  were  once  critical  may  become  of 
little  importance,  or  vice  versa.  All  interdependencies 


between  factors  also  need  to  be  specified. 

Since  intangible  factors  are  the  most  difficult  of 
all  factors  to  quantify  and  the  most  subjective#  it  seems 
appropriate  to  mention  again  sensitivity  analysis.  When 
the  final  estimates  have  been  derived  and  included  in  the 
selection  method#  some  time  should  be  devoted  to  studying 
the  effect  of  varying  the  values  for  the  intangible 
factors  and  watching  the  effect  on  the  final  decision. 
Management  should  be  briefed  on  these  effects  and  the 
conditions  which  will  cause  them#  so  they  can  make  an 
informed  decision. 

One  final  comment  on  reducing  subjectivity#  all 
the  suggestions  mentioned  in  this  section  will  cost  time 
and  money.  Estimates  of  intangibles  will  always  be 
subjective#  so  the  amount  of  time  and  money  spent  to 
improve  the  estimates  should  correspond  to  the  importance 
of  the  factor  to  the  decision  and  the  total  cost  of  the 
project.  This  author  calls  this  the  "Good  Enough  Rule"; 
in  other  words#  you  shouldn't  waste  too  much  time  and 
money  on  the  trivial. 

Measurement  of  Intangible  Factors 
Related  to  Computer  Growth  Stages 


Measurement  of  intangible  factors  can  be  basically 
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divided  into  two  types:  either  a  dollar  figure  is 
attached  to  the  cost  or  benefit/  or  some  sort  of  ranking 
comparison  is  used.  The  type  of  measurement  tool  used  to 
derive  and  report  estimates  of  intangible  factors  to 
management  may  be  linked  to  the  stage  of  computer 
development  within  the  organization.  Smith  developed  this 
idea  of  matching  measurement  tools  to  stages  of 
development  for  measuring  intangible  benefits  of 
information  systems.  [37]  While  his  measurement  tools  are 
not  unique,  their  application  to  intangible  benefits  is 
unique.  This  author  will  enlarge  his  ideas  to  apply  also 
to  intangible  costs  and  adapt  them  for  use  during  the 
computer  selection  stage. 

Smith's  four  stages  of  computer  development  were: 
Initiation,  Expansion,  Formalization,  and  Maturity.  In 
Intiation,  with  the  first  introduction  to  computers, 
management  will  look  closely  at  cost  data  coupled  with  a 
list  of  intangible  benefits.  During  Expansion,  computers 
have  proven  their  worth  and  "...  budgeting  is  loose"; 
managers  may  only  need  a  "benefit  profile"  to  justify 
expenditures.  Rapid  growth  and  generous  budgets  are 
reduced  in  Formalization;  management  is  more  conservative 
and  requires  cost  data  justification  for  expenditures.  In 
the  final  stage,  Maturity,  upper  management  is  more  aware 
of  the  potential  use  of  computers  and  trusts  the  EDP/MIS 


manager  to  guide  the  company  in  "...  terms  of  a  master 
plan  for  information  resource  management."  At  this  stage 
the  "benefit  profile"  is  again  appropriate  for 
presentation  of  intangibles.  137,  pp.26-7J 

If  your  organization  is  in  the  first  or  third 
stage,  then  a  method  of  measuring  intangibles  that  assigns 
cost  data  may  be  appropriate  to  use.  The  assignment  of  a 
dollar  figure  is  very  difficult;  but  may  also  be 
necessary  due  to  the  overall  selection  method  being  used, 
such  as  Cost/Benefit  Analysis,  or  management's  desire  to 
reduce  everything  to  a  dollar-and-cents  decision.  When 
intangibles  must  be  costed,  some  of  the  same  techniques 
used  for  tangible  costing  may  be  useful.  Some  useful 
techniques  are: 

(1)  Expected  values,  using  probability  to  assign 
savings  or  costs,  to  provide  a  better  estimate. 

(2)  Applying  a  discount  rate  to  arrive  at  a  more 
accurate  future  value  of  a  cost  or  benefit. 

(3)  Applying  a  risk  factor  to  reflect  a  benefit  not 
being  achieved  or  a  cost  being  exceeded. 

(4)  Determining  a  payback  period:  period  required 
for  costs  and  benefits  to  be  equal. 

(5)  Determining  development  time  or  time  required  to 
become  fully  operational. 

If  your  organization  is  in  the  Initiation  or 
Maturity  stage,  then  a  ranking  method  of  measurement  such 


as  the  intangible  profile  may  be  most  appropriate  to  use. 
This  author  believes  that  this  approach  for  evaluating 
intangible  factors  is  preferred,  since  intangibles  by 
nature  defy  costing.  Smith  derived  the  "benefit  profile", 
a  comprehensive  checklist  of  possible  benefits  that  could 
be  used  to  justify  an  information  system,  as  part  of  a 
corporate  research  project.  (See  Appendix  E)  The  idea  was 
to  produce  a  master  list  of  possible  intangible  benefits 
to  assure  that  significant  intangible  benefits  were  not 
overlooked.  After  the  applicable  intangible  benefits  have 
been  identified,  the  benefits  can  be  weighted  and  ranked 
for  each  alternative.  (37,  p.26] 

The  same  sort  of  approach  can  be  used  to  measure 
both  intangible  benefits  and  costs.  This  is  the  basis  for 
the  development  of  an  Intangible  Factor  Profile  in  the 
next  chapter. 


CHAPTER  V 


CASE  STUDY 

This  case  study  was  based  on  an  Air  Force  unit 
which  recently  replaced  a  very  large  computer  system  with 
one  from  a  different  vendor.  The  name  of  the  unit  and  the 
people  involved  are  irrelevant  to  the  study,  so  will  not 
be  mentioned.  General  characteristics  will  be  supplied, 
so  the  reader  may  determine  if  this  case  study  is  similar 
to  and  perhaps  applicable  to  his/her  area  of  interest. 

The  ideas  developed  and  the  views  presented  are 
those  of  this  author,  and  may  not  necessarily  reflect  the 
views  of  the  Air  Force  or  Department  of  Defense. 

BacRgiflund  and  Purpose  of  Study 

This  particular  case  study  was  chosen  for  a  number 
of  reasons.  This  system  is  a  very  large  computer  system 
with  multiple  types  of  processing  requirements  and  users. 
This  author  had  firsthand  access  and  experience  with  the 
unit  during  a  portion  of  the  replacement  process.  Most 
importantly,  this  replacement  was  so  complex  and  lengthy 
that  it  seemed  a  fertile  ground  to  discover  many  of  the 


intangible  factors  that  could  later  "come  back  to  haunt" 
the  decision  maker. 

The  original  system  that  was  replaced  consisted  of 
two  mainframes ,  with  five  processors.  The  peripherals 
filled  several  rooms.  Multiple  types  of  processing,  such 
as  batch,  on-line,  and  real-time,  are  supported  by  this 
system.  The  programs  total  over  two-million  lines  of 
code.  Programs  access  records  in  various  types  of  files: 
fixed,  variable,  indexed,  sequential,  and  database. 

Record  sizes  can  range  up  to  thousands  of  characters; 
hundreds  of  thousands  of  records  can  belong  to  one  file. 
The  historical  tapes  range  across  multiple  reels  and  are 
maintained  for  many  years.  The  users  of  this  system  span 
several  buildings  locally  and  extend  to  several  worldwide 
sites. 

The  replacement  process  took  over  ten  years  and 
over  fifty  million  dollars,  to-date.  The  acquisition 
process  alone  took  over  six  years,  twice  as  long  as  the 
total  planned  replacement  time.  Once  "operational",  the 
new  system  took  three  years  of  parallel  processing  support 
before  it  truly  replaced  the  old  system.  The  new  system 
consists  of  seventeen  processors  with  eight  million  words 
of  memory:  these  numbers  address  only  mainframe  capacity 
and  do  not  include  smaller  computers  in  the  distributive 
processing  network.  These  figures  have  been  included  only 
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to  substantiate  the  claim  that  this  was  a  large,  lengthy, 
and  complex  replacement  of  computer  systems. 

This  replacement  of  computer  systems  did  provide 
many  examples  of  problems.  These  problems  often  seemed  to 
be  linked  to  intangible  factors  which  were  not  allowed  to 
become  part  of  the  selection  decision.  This  experience 
inspired  the  author  to  develop  this  thesis.  The  purpose 
of  including  this  case  study  was  to  develop  a  list  of 
intangible  factors  encountered,  especially  the  most  costly 
ones;  and  to  devise  a  method  of  capturing  and  evaluating 
intangibles. 


The  impact  of  ignored  intangible  factors  was 
abundantly  illustrated  as  the  replacement  described  in 
this  case  study  progressed.  Employees  and  managers  alike 
would  gather  and  list  all  the  different  factors  that  were 
combining  to  make  the  change  of  vendors  a  very  costly 
process.  The  new  vendor  had  won  the  contract  by  being  the 
least  costly,  but  new  additional  costs  were  quickly 
rising.  There  were  feelings  running  from  anger  to 
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The  preceding  description  was  the  real  life 
situation  the  author  observed,  but  similar  descriptions 
can  also  be  found  throughout  computer  literature.  While 
this  situation  was  a  trying  time  to  live  through,  it  was 
also  a  valuable  source  of  insight  into  the  importance  of 
inclusion  of  intangible  factors  in  the  selection  process 
and  a  good  source  for  collecting  a  list  of  intangible 
factors. 

While  computer  literature  did  contain  horror 
stories  of  computer  replacements  gone  bad,  it  did  not 
furnish  much  insight  into  how  to  handle  the  problem. 

Today,  industry  seldom  even  considers  large  system 
replacement  with  anything  but  the  current  vendor  or 
compatible  units.  [16,  pp.4-6]  However,  since  government 
acquisitions  will  continue  to  driven  by  the  factors 
explored  in  Chapter  2  of  this  thesis,  the  government  will 
probably  continue  changing  vendors  when  replacing  computer 
systems.  Therefore,  the  handling  of  intangible  factors 
generated  by  a  change  of  vendors  is  important  to 
government  acquisition.  Conversely,  some  of  these 
intangible  factors  may  not  be  of  interest  to  industry. 

intangible  Factor  Profile 

The  development  of  the  Intangible  Factor  Profile 
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(Appendix  F}  was  a  combination  of  factors  gleaned  from 
literature,  interviews  with  experts,  reviews  of  the  case 
study's  documents,  and  personal  experience.  There  is  very 
little  research  on  this  topic,  so  the  bulk  of  the  profile 
comes  from  the  case  study. 

The  profile  is  merely  an  organized  list  of 
possible  intangible  factors.  Its  suggested  use  is  to 
provide  some  examples  of  intangible  factors  that  may  be 
found  during  a  change  of  computer  systems.  The  reader  may 
use  it  to  start  his/her  own  list  of  intangibles  or  may 
want  to  use  it  to  check  after-the-fact  if  anything  was 
missed. 

The  profile  began  as  a  list  of  intangibles  entered 
just  as  they  were  encountered.  The  list  was  growing  long 
and  lacked  organization;  the  need  for  categories  or 
headings  became  apparent.  Literature  provided  a  starting 
point  for  possible  headings;  combinations  of  Robinson's 
[34]  and  Vaid-Raizada ' s  works  [40]  formed  the  original 
headings.  These  were  revised  and  supplemented  by 
information  from  the  case  study's  documents  and 
interviews.  All  the  major  headings  may  not  apply  to  the 
reader's  situation.  (See  Table  5.1)  Again,  these  may  be 
selected  as  appropriate  to  the  reader's  needs. 
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TABLE  5.1 

MAJOR  HEADINGS  PROM  INTANGIBLE  FACTOR  PROFILE 


1. 

Planning  Process 

12. 

Supplies 

2. 

Site  Preparation 

13. 

Communications/ 

3. 

Conversion 

Networking 

4. 

Hardware 

14. 

Future  Expansion 

5. 

Software 

15. 

Delays 

6. 

Systems  Performance 

16. 

Documentation 

7. 

Reliability 

17. 

Employee  Morale 

8. 

Security 

18. 

Personnel 

9. 

Vendor  Support 

19. 

Training 

10. 

Ease  of  Use 

20. 

Risks 

11. 

Maintenance 

21. 

Lost  Opportunities 

TABLE  5.2 

CATEGORY  10  FROM  THE  INTANGIBLE 
FACTOR  PROFILE  [Appendix  F] 

10.  EASE  OF  USE 

-  Degree  of  difference  from  old  system 
—  For  operators 

—  For  systems  programmers 
—  For  other  programmers/analysts 
—  For  end  users 

-  "User  friendly"  rating 
—  For  operators 

—  For  systems  programmers 
—  For  other  programmers/analysts 
—  For  end  users 


A  word  of  warning  -  the  placement  of  individual 
intangibles  under  a  particular  heading  may  not  comply  with 
the  reader's  ideas  of  organization.  Some  individual  items 
were  easy  to  categorize:  "Resistance  to  change"  was  easy 
to  identify  as  belonging  to  "17.  Employee  Morale". 

However ,  most  items  could  easily  belong  to  several 
different  categories.  Where  did  "Degree  of  difference 


from  old  system"  belong?  (See  Table  5.2)  It  could  be 
considered  part  of  "10.  Ease  of  Use",  or  was  it  a 
conversion  problem,  or  maybe  it  belonged  to  training; 
perhaps  it  should  be  a  whole  new  heading. 

This  author  has  heard  about  attempts  to  include 
intangibles  that  were  eventually  abandoned  because 
intangibles  were  just  so  difficult  to  handle.  Don't  be 
discouraged  by  the  indefinite  way  they  categorize.  The 
best  bet  may  be  to  collect  the  individual  items  and  then 
divide  them.  The  central  idea  is  to  make  their  treatment 
more  manageable  and  to  make  sure  each  one  is  accounted  for 
and  not  entered  multiply. 


The  process  of  developing  the  Intangible  Factor 
Profile  led  to  the  development  of  a  worksheet  to  capture 
and  track  information  on  intangibles.  The  idea  behind  the 
Intangible  Factor  Worksheet  (Appendix  F)  was  to  formally 
capture  information  about  intangibles  as  it  was  discovered 
and  to  be  able  to  track  and  account  for  intangibles  during 
the  course  of  the  acquisition  process.  The  idea  is  again 
aimed  at  systematizing  the  treatment  of  intangibles,  as 
discussed  in  Chapter  4  of  this  thesis. 

The  format  of  the  worksheet  was  purposefully 


generalized  to  be  useful  for  either  an  intangible  benefit 
or  intangible  problem.  (The  word  problem  was  used  in  the 
place  of  cost,  which  implies  the  ability  to  assign  a 
dollar  cost.)  The  worksheet's  heading  may  be  modified  to 
match  whatever  categories  are  being  used  by  the  reader's 
selection  process.  (See  Table  5.3)  The  idea  is  to  be 
able  to  sort  the  intangibles  into  manageable  groups  and 
treat  them  much  the  same  way  tangible  factors  are  being 
treated.  Creating  this  worksheet  as  a  database  file  on  a 
small  computer  may  be  a  good  way  to  organize  this 
information.  All  the  intangibles  can  then  be  retrieved, 
printed,  or  checked  by  category  quite  easily. 

TABLE  5.3 

TOP  PORTION  OF  INTANGIBLE  FACTOR  WORKSHEET 

_  BENEFIT  Major  Category  _ 

Sub-Category  _ 

_  PROBLEM  Factor  Number  _ 

Stage/Milestone  _ 

The  second  portion  of  the  worksheet  provides  space 
for  a  title  and  brief  description  of  the  intangible 
factor.  This  may  be  all  the  information  that  is  captured 
when  the  intangible  is  first  discovered,  but  putting  this 
information  on  a  form  that  will  be  dealt  with  later  will 
insure  that  the  intangible  is  not  lost  or  forgotten. 

The  next  question,  "Related  to/Dependent  on  other 
factors?",  is  the  place  to  put  relationships  and 
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assumptions  that  may  change  the  effect  of  the  intangible. 
This  section  is  also  the  key  to  determining  the  answer  to 
"what-if"  types  of  questions  for  intangibles.  For 
example,  in  the  case  study  conversion  from  the  old 
programs,  written  in  a  language  only  supported  by  the  old 
system,  to  new  programs  on  the  new  machine  was  a  $5 
million,  70  man-year  problem.  Contributing  to  this 
problem  were  the  related  intangible  factors  of:  the 
unit's  inexperience  with  the  new  system;  the  inexperience 
of  the  vendor's  on-site  "experts";  the  non-responsiveness 
of  the  vendor's  off-site  experts;  coupled  with  poor, 
late,  and  inaccurate  documentation.  Had  any  one  of  those 
related  factors  been  changed,  perhaps  the  conversion  might 
have  been  less  of  a  problem. 

The  bottom  portion  of  the  Intangible  Factor 
Worksheet  was  designed  for  capturing  information  during 
the  evaluation  or  rating  phase  of  the  selection  process. 
(See  Table  5.4)  Again  this  is  just  an  idea  of  information 
that  might  be  helpful,  design  this  part  to  reflect  the 
type  of  selection  method  being  used  on  your  project.  The 
idea  again  is  to  try  to  build  into  the  form  an  attempt  to 
make  the  estimations  or  rankings  less  subjective;  this 
format  includes  probability  figures  so  expected  values  may 
be  calculated. 


TABLE  5.4 

BOTTOM  PORTION  OF  INTANGIBLE  FACTOR  WORKSHEET 
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Most  likely  to  be:  Chance  of  Dollar  estimate 

occurring  (if  possible) 

_  Slight  _  _ 

_  Average  _  _ 

_  Above  Average  _  _ 

_  Very  Great  Impact  _  _ 

Possible  Impact:  _ 

The  "Dollar  estimate"  column  of  Table  5.4  may  be 

used  to  actually  provide  a  dollar  figure  for  the 

intangible;  this  may  be  necessary  due  to  the  way 

intangibles  are  going  to  be  factored  into  the  selection. 

(Using  Cost/Benefit  Analysis)  Another  way  to  use  this 

estimate  is  if  a  solid  dollar  figure  can  be  applied  to 

this  factor,  then  it  should  be  reclassified  as  a  tangible 

factor  and  treated  accordingly. 

The  "Possible  Impact"  section  of  Table  5.4  was 
included  to  capture  a  brief  statement  when  the  impact  was 
impressive.  This  could  be  a  very  good  benefit  or  a 
possible  disaster  statement.  These  quotes  are  really 
helpful  and  impressive  for  briefings  to  management. 

Validating  and  Quantifying  Profile 

The  Intangible  Factor  Profile  was  validated  in  two 
ways.  It  was  first  reviewed  for  accuracy  by  several 
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experts  working  at  the  case  study  site.  The  profile  was 
left  with  each  reviewer  for  consideration.  It  was  later 
retrieved  during  an  interview  where  the  author  gathered 
further  information.  The  second  validation  was  performed 
using  information  from  the  historical  documents  pertaining 
to  the  case  study.  This  information  was  accumulated  and 
entered  into  the  final  profile.  Both  these  validation 
efforts  were  aimed  at  assuring  that  the  intangibles  listed 
in  the  profile  really  existed. 

The  quantification  step  also  came  from  the  above 
two  sources.  It  was  quite  interesting  that  many 
intangible  factors  started  as  problems  with  no  real 
dollars  attached,  but  often  ended  being  connected  to  lost 
dollars  and  time.  The  two  biggest  problem  areas  were 
conversion  and  delays.  These  were  followed  by 
communications  and  hardware,  and  all  these  seemed  somehow 
related  to  vendor  support. 

Some  examples  of  these  problems  areas  will  perhaps 
give  the  clearest  picture  of  the  impact  of  these 
intangible  factors.  Conversion  was  quoted  as  a 
"monstrous"  problem:  total  cost  was  $10.4  million,  27 
months,  or  95  man-years.  There  were  several  reasons  the 
conversion  couts  grew  to  monstrous  proportions.  (See 


Table  5.5)  The  costs  were  underestimated:  first 


estimates  of  25  man-years  were  based  on  only  five  percent 


.'-■-V'j-  V  V  '.-  V  V'.'  V  .•  • 


78 


of  the  old  code  being  converted,  the  rest  being  done  by  a 
translator.  The  translator  could  not  be  written  and 
finally  had  to  be  dropped  to  allow  competitive  bidding. 
(Remember  Chapter  2's  review  of  government  acquisition!) 


TABLE  5.5 

CATEGORIES  3  &  15  FROM  THE  INTANGIBLE 
FACTOR  PROFILE  [Appendix  F] 

3.  CONVERSION 

-  Degree  of  difficulty 
—  Standard  languages 
—  Special  languages 

-  Possibility  of  being  more  difficult 

-  Possibility  of  taking  more  time  than  planned 

-  Possibility  of  translators  not  working 


15.  DELAYS 

-  Ability  to  manage 

-  Loss  of  confidence 

-  Continued  leasing  of  old  system 

-  Extent  and  possibilities  of  delays 
—  Hardware  late 

—  Software  releases  late 
—  Vendor  not  able  to  fix  problem 

-  Unsuspected  complexities  in  system 

-  Loss  of  trained  staff  (normal  rotation, 
retirement) 


Delays  were  as  big  a  problem  as  conversion,  but 
more  difficult  to  attach  dollar  figures  to  the  intangible 
problems'  results.  (See  Table  5.5)  The  whole  acquisition 
process  was  stretched  out  to  six  years,  often  due  to 
having  to  rewrite  specifications  to  make  the  bids 
competitive.  The  delays  continued  as  the  new  system 


arrived:  new  hardware  and  software  were  either  not  ready 
for  release  on  time  or  did  not  work;  and  unsuspected 
problems  and  complexities  were  discovered.  This  all 
resulted  in  the  continued  leasing  of  the  old  system  for 
another  three  years,  a  much  longer  time  than  ever 
anticipated.  There  were  other  spin-offs  of  the  delays; 
such  as  loss  of  the  staff  that  had  been  trained  in  the  new 
system,  due  to  normal  military  rotation,  before  the  new 
system  was  ever  available  for  use. 

Since  the  acquisition  process  took  so  long,  there 
were  many  needs  that  had  developed  since  the 
specifications  for  the  replacement  computer  had  been 
written.  One  especially  troublesome  area  was  the  needed 
ability  to  transmit  information  at  a  greater  speed  than 
was  written  into  the  specifications.  The  new  computer  met 
specifications,  but  did  not  have  the  technology  to  meet 
the  true  need.  It  also  could  not  be  joined  into  the 
needed  distributive  network  with  other  of  its  smaller 
machines,  because  they  were  not  really  compatible. 

The  hardware  for  the  new  system  was  another 
problem  area.  Available  memory  was  misleading  since  it 
was  not  usable  as  a  virtual  machine.  The  amount  of 
storage  on  disk  packs  was  also  misleading  because  of  the 
restrictions  placed  on  their  use.  The  records  and  files 
of  the  existing  system  were  so  large  that  they  could  not 


be  easily  handled  on  the  new  system.  It  was  later 
discovered  that  the  benchmark  size  of  files  had  been 
reduced  because  no  vendor  wanted  to  commit  that  amount  of 
hardware  to  a  full-blown  test. 


TABLE  5.6 

CATEGORY  9  FROM  THE  INTANGIBLE 
FACTOR  PROFILE  I Appendix  F] 

9 .  VENDOR  SUPPORT 

-  Vendor  ability  to  meet  deadlines 

—  Hardware  delivery  &  installation 
—  Software  modifications 
—  Training 

-  Vendor  reputation 

-  Stability 

-  History 

-  Years/degree  of  expertise 
—  Hardware 

—  Software 
—  Maintenance 

-  Knowledge  of  vendor's  staff 
—  Training/teachers 

—  On-line  help 

—  On-site  technical  representatives 
—  Off-site  technical  representatives 

-  Easy  to  get  help 

—  Type  of  communication  (mail  form 
VS  "800"  line) 

—  Speed  of  response 


The  one  area  that  had  more  individual  intangible 
entries  than  any  other  was  vendor  support.  (See  Table 
5.6)  This  one  factor  is  often  cited  by  industry  as  the 
main  reason  an  upgrade  of  computers  is  made  within  the 
current  vendor's  line.  Many  of  the  problems  mentioned 
before  could  have  been  more  easily  overcome  with  good 


vendor  support.  The  vendor's  staff  needs  to  be 
technologically  sound,  have  several  years  of  experience 
with  the  products,  be  able  to  understand  and  explain 
problems,  and  most  importantly,  be  available!  The  more 
difference  there  is  between  the  current  system  and  the  new 
system,  the  greater  the  weight  that  should  be  applied  to 
vendo r  suppo r t . 

This  author  could  continue  illustrating,  with  even 
greater  detail,  the  problems  listed  individually  in  the 
Intangible  Factor  Profile;  however,  the  purpose  of  this 
section  was  just  to  convince  the  reader  that  these 
intangible  factors  were  real  and  had  a  great  impact.  If 
the  reader  has  not  yet  been  convinced,  then  the  author 
suggests  checking  with  other  "systems  folks"  -  everyone 
seems  to  have  their  own  story  quite  similar  to  this  one. 


The  Intangible  Factor  Evaluation  sheet  in  Appendix 
F  was  developed  by  the  author  as  way  to  consolidate  and 
report  the  final  results.  The  evaluation  method  suggested 
by  this  author  is  basically  a  weighted  score  method,  with 
a  few  new  twists.  (See  Chapter  3  of  this  thesis  for 
presentation  of  three  weighted  score  methods.)  This 
method  was  selected  because  ratings  seem  to  be  a  better 
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way  to  compare  intangible  factors.  If  it  was  easy  to 
attach  dollar  figures  to  intangibles ,  they  would  not 
really  Jbe  intangibles. 

One  new  suggestion  is  to  implement  the  basic 
weighted  score  method  using  spreadsheets  on  a  computer  for 
the  evaluations.  The  sample  spreadsheet  in  Appendix  F 
shows  how  intangible  factors  may  be  grouped  into  major 
headings  to  provide  a  total  comparison  figure  for  each 
vendor.  The  numbers  in  this  spread  sheet  can  be  varied 
and  the  effect  reported.  This  is  an  easier  way  to  do 
sensitivity  analysis.  Each  major  heading  can  also  have 
its  own  spreadsheet  composed  of  its  individual  intangible 
factors.  This  will  aid  in  computing  values  for  the  main 
evaluation  and  allow  easy  sensitivity  analysis  at  the 
individual  factor  level. 

Another  suggestion  is  to  choose  a  natural  rating 
range  with  which  the  evaluators  are  familiar.  This  author 
used  a  "grading"  range:  0  -  100.  This  approach  allows 
more  than  one  factor  to  receive  a  grade  of  95  and  the 
final  evaluation  ratings  are  adjusted  automatically  by  the 
spreadsheet.  It  would  be  helpful  to  also  provide  the 
evaluators  with  a  standardized  guide  defining  what  a  ”95" 
score  means ,  so  it  means  the  same  to  all  evaluators.  This 
equalization  of  scoring  can  be  discussed  and  trial  ratings 


be  done  by  gathering  the  evaluators  together  before  any 


actual  systems  are  rated.  The  real  ratings  should  be 
completed  separately  though,  as  discussed  in  Chapter  4  of 
the  thesis. 

The  final  use  of  these  composite  rating  scores 
depends  upon  the  method  of  comparison  being  used  for 
selection.  These  total  intangible  ratings  could  be  used 
to  divide  the  total  cost  estimates  for  each  system,  to 
arrive  at  a  Weighted  Evaluation  of  Cost  Factors.  (See 
Chapter  3)  Another  possibility  would  be  the  combination 
of  these  ratings  with  ratings  for  tangible  factors.  A 
third  possibility  might  be  to  treat  intangibles  as  a 
separate  section  when  reporting  all  the  facets  of  the 
selection  process. 

Whatever  approach  is  used  to  report  and  include 
intangibles,  it  is  important  that  assumptions  and 
dependencies  are  also  reported.  The  results  of 
sensitivity  analysis  are  al?o  important,  especially  when 
coupled  with  estimations  of  the  likelihood  of  the  ratings 
being  correct.  The  bottom  line  is  that  selection  of 
something  as  important  as  a  large  computer  system  should 
not  be  reduced  to  looking  at  some  final  magic  numbers. 

All  the  final  magic  numbers  have  lots  of  subjectivity  and 
estimation  built  in.  The  managers  are  paid  to  use  their 
judgment  and  make  good  decisions:  all  this  analysis  is 


not  sacred,  but  just  good  inputs  to  be  used  for  making 
that  final  decision. 
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Several  assumptions  were  made  by  this  author  in 


the  development  of  the  profile,  worksheet,  and  evaluation. 


First  it  was  assumed  that  it  would  still  be  possible  to 


capture  intangible  factors  important  to  the  selection 


stage,  after  the  replacement  was  completed.  It  was  also 


assumed  that  a  rating  method  was  the  best  way  to  evaluate 


intangible  factors.  Finally,  the  author  has  assumed  that 


the  value  of  publishing  a  collected  list  of  intangibles  is 


that  the  captured  intangibles  will  apply  equally  well  to 


other  similar  projects. 


Conversely,  a  major  limitation  of  the  study  may  be 


that  the  government  procurement  constraints  may  generate 


intangible  factors  that  are  unique  to  that  environment. 


Another  limiting  factor  was  the  loss  of  the  opportunity  to 


interview  some  key  case  study  personnel  due  to 


reassignment  or  retirement.  The  study  is  further  limited 


by  the  fact  that  these  suggested  methods  of  treating 


intangibles  have  not  been  applied  during  a  selection 


situation. 


CHAPTER  VI 


CONCLUSIONS  AND  RECOMMENDATIONS 

This  thesis  has  provided  an  overview  of  government 
ADP  procurement,  a  review  of  six  current  computer 
selection  methods,  plus  methods  and  tools  to  use  on 
intangible  factors  important  to  the  selection.  Some  of 
the  ideas  were  not  new  but  being  able  to  find  them 
compiled  within  one  source  is  new.  The  methods  and  tools 
to  use  on  intangibles  are  unique,  and  little  other  work  is 
available  on  this  topic  at  this  date. 

Hopefully,  this  work  will  be  of  value  to  the 
analyst/manager  needing  this  background  information  to 
start  the  difficult  process  of  selecting  a  large  computer 
system's  replacement. 


Lessons  Learned 

The  heading  for  this  section  comes  from  this 
author's  Air  Force  background.  "Lessons  Learned"  are  the 
standard  heading  for  collecting  all  the  things  you  want  to 
do  differently  if  you  did  the  same  activity  again  or  maybe 
just  newly  discovered  realities.  There  were  several 
lessons  learned  during  the  course  of  this  study. 


The  author  was  already  aware  of  many  differences 
between  government  procurement  and  the  way  industry  does 
business ,  but  reviewing  the  literature  revealed  just  how 
different  the  approaches  could  be.  For  example,  industry 
has  no  problems  buying  "sole  source",  while  this  way  of 
buying  is  not  accepted  by  government  for  the  scale  of 
purchase  described  in  the  case  study.  There  seems  to  be 
justification  for  buying  a  large  computer  system 
replacement  in  this  manner  from  the  study  of  literature 
and  this  case  study.  The  intangible  factors  seem  to  be  a 
driving  force  in  industry's  decision  to  replace  large 
computer  systems  with  the  same  vendor;  perhaps  government 
will  be  able  to  adopt  this  stand  in  the  future. 

The  author  also  had  suspected  that  at  the  end  of 
the  thesis  that  one  method  of  selection  would  stand  out  as 
"The  Way"  to  do  business.  Instead,  the  following  methods 
selected  and  presented  for  review  in  Chapter  3: 
Cost/Benefit  Analysis,  Weighted  Score,  Payoff  Matrix, 
Cost-Value,  Interactive  Financial  Planning,  and  Weighted 
Evaluation  of  Cost  Factors,  were  all  useful.  There  are 
even  more  good  approaches,  such  as  the  ELECTRE  and 
Brocato's  method,  that  were  not  reviewed!  No  panacea 
exists  for  the  analyst/manager  having  to  develop  a  way  to 
select  among  competing  vendors.  Selecting  the  right 
method  is  driven  by  a  multitude  of  factors,  ranging  from 


the  analyst's  experience  to  "what  the  boss  will  buy". 

A  final  lesson  learned  was  that  intangibles  really 
£jl£  difficult  to  handle.  The  author  also  had  a 
preconceived  notion  that  a  little  research  effort  would 
reveal  a  way  to  quantify  and  evaluate  intangibles.  There 
really  is  no  way  to  fix  dollar  values  on  everything; 
therefore,  the  decision  makers  simply  must  be  convinced  to 
include  these  intangible  factors  into  the  decision  even 
though  dollar  figures  may  not  be  attached.  Chapter  5 
suggested  three  ways  to  incorporate  a  composite  rating  for 
intangibles  into  the  decision:  Weighted  Evaluation  of 
Cost  Factors,  combined  with  ratings  for  tangibles,  or  a 
separate  report  section. 


The  suggestions  of  how  to  treat  intangible  factors 
during  the  selection  process  developed  in  Chapters  4  and  5 
need  to  be  subjected  to  further  research.  Specifically, 
they  need  to  be  validated  by  use  during  a  large  computer 
replacement  project  and  implemented  from  the  beginning  of 
the  project. 

Similar  studies  need  to  be  conducted  to  add  to  the 
Intangible  Factor  Profile's  list  of  individual  items. 
Perhaps  more  research  could  eventually  compile  a 
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comprehensive  list  of  factors  that  the  analyst  could  be 
assured  was  valid. 

More  research  is  also  needed  to  develop  a 
selection  process  that  is  able  to  discover  the  mix  of 
dollar  figures  and  intangible  figures  that  result  in 
making  the  best  decision. 

Finally,  perhaps  further  research  will  modify  the 
government's  accepted  ways  of  replacing  large  computers. 
Perhaps  intangible  factors  will  be  accepted  as  a  driving 
force  in  the  selection  process. 
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Tangible,  Equipment,  Other  cost  center  charges  and  benefits.  Allocated  charges  based  on  square  feet, 

indirect  software,  etc.  general  and  administrative  expenses  and  number  pf  personnel,  direct  expenses, 

benefits  number  of  departments 
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Total  for  other  items  .  .  304  219  417 

Total  for  each  vendor  .  627  560  663 


APPENDIX  C 


MARTIN'S  PAYOFF  MATRIX  CHECKLIST 


PAYOFF  MATRIX 

(Decision  Making  Checklist) 


Develop  Alternatives. 

Determine  States  of  the  Future. 

a.  What  is  Planning  Horizon? 

b.  Which  states  are  more  relevant? 

Determine  Payoff  Values. 

a.  Determine  Selection  Factors. 

b.  Assign  relative  weights  to  factors. 

c.  For  each  factor. 

(1)  Determine  subfactors. 

(2)  Allocate  factor  weight  to  subfactor  according 
to  relative  importance. 

(3)  For  each  state  of  the  futurer  rank 
alternatives  for  each  subfactor. 

(4)  Compute  weighted  ranking,  for  each  subfactor. 

(5)  Sum  weighted  rankings  to  arrive  at  weighted 
total  for  each  factor. 

d.  Sum  weighted  totals  for  each  factor  to  arrive  at 
Grand  Weighted  total  which  is  the  payoff  value. 

Solve  the  Payoff  Matrix. 

Select  the  Best  Alternative. 

Perform  Sensitivity  Analysis. 

Go  Back  to  Steps  2  or  3  as  necessary. 


.  -a  i  ■  v*  -vr  >-»* 
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BROCATO'S  MASTER  LIST 
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BROCATO’S  MASTER  LIST  [30,  pp.52-3) 


Hardware  , 

Central  Processor 

-  Instruction  set  and  special  features  (flexibility  and 

power  of  the  instruction  set,  availability  and  fexibility  , 

of  the  decimal  instruction  set,  ease  of  bit  manipulation). 

-  Addressing  (amount  of  directly  addressable  core,  virtual 
memory,  indirect  addressing). 

-  Double-precision  arithmetic  functions. 

-  Availability  of  storage-to-storage,  storage-to-register 
and  register-to-register  instructions. 

-  Fetch  time  and  cycle  time. 

-  Size  (words  in  memory,  word  size) . 

-  Input/output  (channel  speed,  spooling,  number  of 
channels,  symbionts  such  as  Hasp,  channel  overlap). 

-  Operator  dependence  (requirements  for  operator 
intervention,  set-up  time) . 

-  Registers  (general  registers,  index  registers, floating 
point  registers,  several  complete  sets  of  registers). 

Eeiiph-erals 

-  Direct-access  storage  (transfer  rate,  speed  of  access, 
maximum  storage  size,  ease  of  changing  storage  elements). 

-  Mass  storage  (transfer  rate,  speed  of  access,  maximum 
storage  size) . 

-  Magnetic  tape  (speed,  density,  number  of  units,  number 
of  tracks,  operator  dependence). 

-  Paper  tape  (speed,  ease  of  loading,  operator  dependence, 
number  of  tape  levels,  tape  width). 

-  Card  punch  (speed,  number  of  stackers,  operator 
dependence) . 

-  Card  reader  (speed,  ease  of  operation,  operator 
dependence) . 

-  Printer  (speed,  character  set,  ease  of  loading  paper, 
fine  adjustments,  operator  dependence,  quality  of  print, 

ease  of  changing  character  set).  , 

-  Communications  equipment  (speed,  number  of  possible 
terminals,  error  rate,  error-detection  techniques, 
error-correction  techniques). 

-  CRT  terminal  (speed,  buffer  size,  remote  distance 
without  communications  drivers,  character  set,  resolution, 
number  of  terminals,  ease  of  operation,  quality  of 
display,  brightness,  color,  persistence). 

-  Optical  character  reader  (speed,  ease  of  operation, 
operator  dependence). 
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-  Magnetic  character  reader  (speed,  operator  dependence, 
ease  o£  operation) . 

-  Incremental  plotter  (on-line  speed,  off-line  speed  to 
generate  plotter  tape,  throughput  speed,  ease  of 
operation,  operator  dependence) . 


-  Priority  interrupts  (hardware  servicing,  software 
servicing,  speed  of  service,  availability  of  priority 
levels) . 

-  Parallel  input  (number  of  parallel  input  terminals, 
built-in  multiplexing,  speed  of  service). 

-  Parallel  output  (number  of  terminals,  multiplexing, 
speed  of  service) . 

-  Control  pulses  (availability,  decode  requirements). 

-  Clocks  (availability,  real-time,  access  by  user). 


Software 

Systems 

-  Operating  system  (core  requirements,  ease  of  use, 
accessibility  and  ease  of  modification,  diagnostic, 
real-time  monitor,  batch  monitor,  time-sharing  monitor, 
input/output  support,  data  protection  in  event  of  power 
failure,  allowing  time-share  users  to  share  programs  in 
core,  allowance  for  altering  nuclei,  auxilary  storage 
requirements  for  operating  system,  size  of  partition 
during  multiprogramming,  data  management  facilities). 

-  General  support  programming  (job  control  language, 
procedure  library,  function  library,  utility  programs, 
assembler,  Fortran  compiler,  Cobol  compiler,  Algol 
compiler,  various  other  compilers,  linkage  editor). 


-  Assembly  language  (execution  times,  ease  of  programming, 
ease  of  debugging) . 

-  Fortran  (level,  special  features,  diagnostics). 

-  Cobol  (level,  special  features,  diagnostics). 

-  Other  user-level  languages  (report  generation, 
sort/merge,  Basic,  linear  programming,  simulation,  Algol, 
etc. ) . 

-  Real-time  (language,  interrupt  servicing). 

-  Time-sharing  (software  servicing). 

-  Communications  (software  servicing). 

-  Compatibility  (with  existing  system,  reprogramming 
requirements,  retraining  requirements). 


‘WWW? 
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Expandability 

-  Core  (availability,  addressability,  size,  ease  of 
modification) . 

-  Hass  storage  (maximum  size,  speed,  ease  of  addition, 
access  time). 

-  Software  (ease  of  modification  of  software  to  support 
hardware  expansions). 

-  CPU 


General , Support 

-  Periodic  maintenance  (frequency,  time  required). 

-  Emergency  service  (hours  available,  location  of  service 
center,  availability  of  service  personnel,  response  time 
to  service  request). 

-  Documentation  (clarity,  how  extensive,  availability  of 
manuals) . 

-  Initial  training  (where  given,  how  extensive,  limit  on 
personnel) . 

-  Future  training  (where  given,  how  extensive,  limit  on 
personnel). 

-  Availability  of  systems  assistance. 

-  Availability  of  local  backup  computer  (at  least  for 
batch  work) . 

-  Availability  and  vendor  support  of  common  user  groups. 

-  Responsiveness  of  vendor  to  technical  questions 
concerning  the  evaluation  (both  the  timeliness  and 
accuracy  of  the  response  should  be  considered  here  and 
this  should  be  a  fairly  high  percentage  weighted  item  in 
the  evaluation. 


Jfrpeiience  of  the.  Vendor 
-  Real-time  data  acquisition.  -  Time-sharing. 


-  Remote  batch. 

-  Telecommunications. 

-  Multiprocessing. 


-  Local  batch. 

-  Multiprogramming. 

-  Simulation. 


APPENDIX  E 
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BENEFIT  PROFILE  CHART 


Intangible  Benefit*  of  EDP/MIS  Leading  to  Improved  Business  Performance 


Business  Function/Activity 


Degree  of  Performance  Improvement 


Significant 
Improvement  Significant 
in  Existing  New 
System  Benefit 


Engineering/Research 


raphlcs 


Interactive  problem  solvin 


within  compan 


with  customer 


Stimulation  of  new  ideas  (e 


Faster  design  (e. 


Control  of  specif lcatlons/drawlngs 


Access  to  technical  information 


Processing  of  engineering  change  orders 


Manpower /pro lect  management 


Management  of  professional  s  time 


reduced  clerical  workload 


Finance /Account  in 


Budget  preparation 


Use  of  operating/leverage 


Privacy  of  data/  informat  ion 


Security  of  data/lnformatlon 


Integrity  (accuracy  of  data 


Planning  &  control  of  liquid  assets 


Capital  budgetln 


Auditing  and  internal  control 


Simplified  reportln 


Timely  reports 


Employee  Relations/Human  Resources 


Identification  of  best  performers 


individual  and  grou 


Strategic  manpower  plannln 


Places  and  methods  for  recrultin 


Improved  government  reportln 


Monitoring  of  EEO.  ERISA.  OSHA  standards 


Higher  motivation  of  work  force 


career  planning  capabllit 


turnover /absenteeism 


fringe  benefits  planning/control 


ob  satisfaction 
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Business  Function/Activity 


Employee  Relatlons/Human  Resources  (cont.) 


Degree  of  Performance  Improvement 


Some 
Improvement 
In  Existing 
System 


Significant 
Improvement 
in  Existing 
System 


Significant 

New 

Benefit 


Employee  training _ 

(e.g..  computer-assisted  instruction) 

Managerial /professional _ 

Understanding  of  how  HRM  functions _ 

Labor  negotiation  capability _ 

Wage  and  salary  planning/ control _ 


Operations/Production 

Shop  floor  control _ 

Production  scheduling _ 

Increase  labor  productivity _ 

Pinpoint  yicld/quallty  problems  faster 
Reduced  non-productive  time  for  supervisors 

Measure  and  report  trends _ 

Accurate  labor  standards _ 


General  Management 


Increased  coiwnunlcatlon  among  departments 

Planning  data  more  qulckly/easlly  accessible _ 

Ability  to  provide  specialised  (what  If)  reports _ 

Faster  development  of  new  systems 

Easier  to  use  system _ 

Increased  secretarial  efficiency/effectiveness 

(e.g. .  word  processing/ text  editing  additions) _ 

Better  meetings _ 

Provides  greater  reliability  (backup) 

Cross  referencing  of  files _ 

Improved  accuracy,  conciseness,  timeliness,  relevance 

of  all  information _ 

Cost  avoidance  (as  opposed  to  performance  improvement) 

precludes  need  to  hire  new  people _ 

need  fewer  computer  programs  (e.g..  database) _ 

need  less  program  maintenance _ 

reduced  communication  charges  (e.g..  distributed  sys 
less  line  charges) 

reduced  travel  cost  (e.g.,  teleconferencing) _ 

better  use  of  programmer  time  (e.g..  on-line  testing) 
Decision  support  Bystem 
toal  seeking 
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IS 


INTANGIBLE  FACTOR  PROFILE 

1.  PLANNING  PROCESS 

-  Level  of  complexity 

2.  SITE  PREPARATION 

-  Possibility  of  being  inadequate 


& 


Lr- 


3.  CONVERSION 


4. 


-  Degree  of  difficulty 
—  Standard  languages 
—  Special  languages 


-  Possibility  of  being  more  difficult 

-  Possibility  of  taking  more  time  than  planned 

-  Possibility  of  translators  not  working 


HARDWARE 

-  Technical  evaluation 

-  Available  VS  usable  resources 

-  Expansion  potential 

-  Compatability  with/similarity  to  old  system 


5.  SOFTWARE 


-  Reliability  &  usability  of  software  packages 
purchased 

—  Certification  of  software 

—  Failure  rate 

—  Other  users’  evaluations 
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-  Ease  of  modifying  purchased  software  for 

—  Operating  system 

—  Pile  management 

—  Security 

—  Library  routines 


6 .  SYSTEMS  PERFORMANCE 

-  Comparison  of  total  times  to  get  job  done 

-  Effort  required  to  optimize 


7.  RELIABILITY 

-  Equipment 

-  Software 


8.  SECURITY 

-  Evaluation  of  security 
—  User  access 

—  File  protection 
—  Program  protection 

-  Ease  of  adapting  to  needs 


9 .  VENDOR  SUPPORT 

-  Vendor  ability  to  meet  deadlines 

—  Hardware  delivery  &  installation 
—  Software  modifications 
—  Training 

-  Vendor  reputation 

-  Stability 

-  History 

-  Years/degree  of  expertise 
—  Hardware 

—  Software 
—  Maintenance 
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Knowledge  of  vendor's  staff 

—  Training/teachers 

—  On-line  help 

—  On-site  technical  representatives 

—  Off-site  technical  representatives 

Easy  to  get  help 

—  Type  of  communication  (mail  form  VS  "800"  line) 

—  Speed  of  response 


EASE  OF  USE 

-  Degree  of  difference  from  old  system 
—  For  operators 

—  For  systems  programmers 
—  For  other  programmers/analysts 
—  For  end  users 

-  "User  friendly"  rating 
—  For  operators 

—  For  systems  programmers 
—  For  other  programmers/analysts 
—  For  end  users 


11.  MAINTENANCE 

-  Evaluation  of  diagnostic  software 


12.  SUPPLIES 

-  Ease  of  obtaining 

1 3 .  COMMUN ICAT IONS/NETWORK ING 

-  Immediate  future  needs  already  available 

-  Speed  of  transfer 

-  Impact  of  proposed  networks  not  working 

-  Impact  of  family  of  computers  not  compatible 


•.V.V.V.'.vAV 


14.  FUTURE  EXPANSION 


-  Systems  performance  VS  future  growth  projections 

-  Ease  of  expansion 

—  Physical 

—  Technical 


15.  DELAYS 

-  Ability  to  manage 

-  Loss  of  confidence 

-  Continued  leasing  of  old  system 

-  Extent  and  possibilities  of  delays 
—  Hardware  late 

—  Software  releases  late 
—  Vendor  not  able  to  fix  problem 

-  Unsuspected  complexities  in  system 

-  Loss  of  trained  staff  (normal  rotation,  retirement) 


16.  DOCUMENTATION 

-  Ease  of  use 

—  Written  to  reader's  level  (end  users  VS 
technicians 

—  Similarity  to  current  format 
—  Real  examples  included 

-  Available  for  everything 

-  Current  and  accurate 

—  Matches  real  performance 

-  Update  procedure 
—  How  handled 
—  Frequency 


-  Delivered  on  time 
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17.  EMPLOYEE  MORALE 

-  Measurement  of  morale 

-  Resistance  to  change 

-  Evaluation  of  technology 


18.  PERSONNEL 

-  Level  of  staff  expertise  with  proposed  vendor 
—  For  operators 
—  For  systems  programmers 
—  For  other  programmers/analysts 
—  For  end  users 


19.  TRAINING 

-  Quality  of  vendors  training 
—  Staff's  knowledge/ability 
—  Self-teaching  aids 

-  Vendor  ability  to  adapt  training 
—  Content 

—  Schedules 
—  Location 


20 .  RISKS 

-  Needing  more  equipment  than  projected 

-  New  or  untested  technology  failing 


21.  LOST  OPPORTUNITIES 

-  New  projects  delayed  by  conversion 


BENEFIT 


_  PROBLEM 


Title: 


Major  Category 
Sub-Category 
Factor  Number 
Stage/Milestone  _ 


Brief  Description: 


Related  to/Dependent  on  other  factors? 


Most  likely  to  be:  Chance  of 

occurring 

_  Slight  _ 

_  Average  _ 

_  Above  Average  _ 

_  Very  Great  Impact  _ 

Possible  Impact:  _ 


1 

1 

1 

i. 

Intangible  Factors 
Major  Headings 

1 

1 

1 

i 

Importance 

Value 

! 

1 

1 

1 

Adjusted 
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To  Create: 

-  Enter  row  headings  in  column  1. 

-  Column  2  contains  the  natural,  grading  type  of  ratings. 

-  Sum  col.  2  and  divide  each  item  in  col.  2  by  the  sum. 

-  Enter  these  figures  in  column  3. 

-  Rate  each  system  and  enter  the  ratings  in  column  4  and 

6,  respectively. 

-  For  each  system,  multiply  value  in  column  3  times  the 

Evaluation  Rating.  Divide  by  100.  Enter  result  in 
Weighed  Evaluation  rating  column. 

-  Example  for  Ease  of  Use,  System  A: 

(22.73  x  90  )/  100  -  20.46 

-  Sum  the  Weighed  Evaluation  Ratings  for  each  system. 


Perform  Sensitivity  Analysis:  Experiment  with  varying  the 
Importance  Values  to  see  if  the  "best"  system  changes. 

Also  try  varying  the  individual  Evaluation  Ratings;  enter 
probable  extremes  and  let  the  spreadsheet  readjust  the 
figures. 


Note  1:  Create  this  type  of  format  on  a  spreadsheet. 
Column  1  shows  major  headings  but  the  same  idea  can  be 
used  for  each  one  of  the  major  headings  to  summarize  the 
individual  factors. 


Note  2:  These  figures  are  for  illustration  only  and  do 
not  represent  true  results  of  any  rating. 
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